Why do so many incompetent men become leaders? This question, which the psychologist Tomas Chamorro-Premuzic uses as the title of his new book, seems designed to provoke (and he’ll definitely get accused of “virtue signalling” because of it). But it’s a subtler matter than you might think. The obvious answer, of course, is sexism. But that can’t be the whole story. Nobody doubts that there are at least some competent men – and presumably a sufficient number of them to lead the world’s corporations and governments. So even if the world is horribly sexist, why aren’t those men running it? In most walks of life, sexism lowers the bar that men must clear in order to succeed. But you’d still expect the talented ones to clear it more easily than the idiots. And yet: look around.

The answer emerging from Chamorro-Premuzic’s work is that there are two things going on here. One is sexism; but the other is a general difficulty in selecting competent leaders, of either sex, over incompetent ones. We habitually interpret traits such as overconfidence and self-absorption as signs of high ability, though in fact they’re negatively correlated with it. They “should be seen as red flags”, but “instead, they prompt us to say, ‘Ah, there’s a charismatic fellow! He’s probably leadership material.’” Men come to dominate the upper echelons because they’re more likely to manifest such traits. Research does also confirm that women are more harshly judged for displaying confidence. But the striking point here is that even a perfectly nonsexist hiring committee, or electorate, would still end up promoting a disproportionate number of mediocre men, provided they continued to take bluster and self-belief as proof of talent. (The book goes on to explore those traits, such as emotional intelligence, calmness, resilience and self-awareness, that actually do indicate an effective leader.)

This way of viewing things also lets us put aside, for now, the endless debate about nature versus nurture, and how far – if at all – genetics might play some role in the fact that more men than women gravitate toward certain careers and positions. Wherever you stand on all that, you surely prefer competent people over incompetent ones; we can all agree that leaders should be selected on the basis of real leadership ability. And if we got better at doing that, Chamorro-Premuzic’s logic implies, we’d end up with far more women in top jobs anyway – not to mention more competent men, since they currently lose out to the narcissistic blowhards, too.

All of which suggests a problem with the advice directed at women to “lean in”, display ambition, and advertise their abilities, in order to push back against workplace bias and rise through the ranks: it risks being an exhortation to women to get better at mimicking the dysfunctional traits of precisely those men who make the worst leaders. Perhaps it would be progress of a sort to replace half the world’s terrible male presidents and crappy male bosses with equally crappy female ones. But it would surely be best of all to replace the crappy people with effective people – a shift that would necessitate the elevation of many more women. Just don’t expect our current crop of leaders to want to lead the revolution.

oliver.burkeman@theguardian.com

Watch this

In a recent TEDx talk, Tomas Chamorro-Premuzic explains our tendency to rate leaders using exactly the wrong criteria.