The most important, and yet least examined, determinant of success or failure – at work and in life – is self-awareness. The ability to understand who we are, how others see us, and how we fit into the world.

Plato instructed us to “know thyself”, while psychologists have argued that this skill is at the core of human survival and advancement. For millions of years, the ancestors of Homo sapiens evolved painfully slowly. But as the neuroscientist VS Ramachandran explains in his book The Tell-Tale Brain, about 150,000 years ago there was an explosive development in the human brain where, among other things, we gained the ability to examine our own thoughts, feelings and behaviours, as well as to see things from another’s point of view. Not only did this transformation create the foundation for art, spiritual practices and language, it came with a survival advantage for our ancestors, who had to work together in order to survive.

Though we may not face the same day-to-day threats to our existence, self-awareness is no less critical. There is strong scientific evidence that people who know themselves and how others see them are happier. They are smarter, superior students. They raise more mature children. They are also tend to be more creative, confident and less aggressive.

But for most people it is easier to choose self-delusion over the cold hard truth. Our increasingly “me” focussed society makes it easier to fall into this trap. Recent generations have grown up in a world obsessed with self-esteem, constantly being reminded of their special qualities, and it is fiendishly difficult to examine objectively who we are and how we’re seen.

My research shows that while 95% of people think they are self-aware, the real figure is closer to 10-15%. Not only are our assessments often flawed, we are usually terrible judges of our own performance and abilities – from leadership skills to achievements at school and work. What’s scary is that the least competent people are usually the most confident in their abilities.

How can we avoid this fate? We must work on two specific types of insight. Internal self-awareness, an inward understanding of our passions and aspirations, strengths and weaknesses and so on. And external self-awareness, knowing how others see you, means understanding yourself from the outside in.

It would be easy to assume that someone with internal self-awareness would also be externally aware – that being in touch with our feelings and emotions helps us tune into how we’re seen. Strangely, research has often shown no relationship between the two – some studies have even revealed an inverse one. Though this isn’t so puzzling when we think about the people we know. That person who spends hours trying out different therapies, but still have no idea that others see them as self-absorbed rather than self-aware.

For those looking to gain true insight, remember that other people often see us more objectively than we see ourselves and that introspection and self-examination can have hidden pitfalls that make insight virtually impossible.

Ultimately there are two types of people – the ones who think they’re self-aware and the few who actually are. It can take a lifetime to move from the former to the latter.

Insight: the Power of Self-Awareness in a Self-Deluded World by Tasha Eurich has just been published by Macmillan at £18.99. To order a copy for £16.99, go to bookshop.theguardian.com