Nobody could have predicted that Keir Starmer’s waistline would have become one of the talking points of budget week. But a throwaway comment by Peter Mandelson, the Labour peer, that he could do with shedding a few pounds paved the way for the chancellor, Jeremy Hunt, to poke fun at him in the Commons, and for shadow ministers to be quizzed over his weight.

Mandelson’s comments came in the context of a distinctly less personal critique of the dress sense of former Labour leaders; he recounted how he bought Tony Blair a collection of expensive, bright ties to project “boldness and confidence”. Ties are one thing, weight another: it’s rude to so publicly opine on a politician’s size, not that Starmer could really be labelled overweight.

But, like it or not, his general observation that appearance matters in politics is correct, and one no experienced adviser ignores. Of course, smartening up or losing excess weight won’t be enough to transform an unpopular leader into a popular one. But however unfair it is, the extent to which a politician is conventionally attractive plays into public perceptions.

A study of gubernatorial elections in the US found that people’s rapid judgments of competence, based solely on the facial appearance of candidates, correctly predicted election outcomes much more than if appearance didn’t matter. Depressingly, experiments suggest that the faces perceived as most politically competent are more masculine as well as more attractive and mature.

It’s not just politics: on a media training course I did 20 or so years ago, we were told that what people take away from an interview is 90% what you look like, 10% what you say; a rule of thumb I can well believe. But the so-called “beauty premium” goes much further than the careers where appearance might be considered important: plenty of studies suggest that the more conventionally attractive someone is, the more likely they are to be paid more, get a job interview, and even get a more favourable outcome in a criminal trial.

This plays out differently according to sex; women are 16 times more likely to perceive they have been discriminated against based on their weight than men, for example, and research based on body scans of people’s overall shape suggests that, for men, it is being taller that pays off, whereas for woman it is having a lower body mass index. Women who show the visible signs of ageing are judged far more harshly than men.

The immediate question is: why? Is it because we’re innately attracted to and value people who are easy on the eye; is it because of the “halo effect” that occurs when our brains unfairly use appearance as a heuristic for other desirable traits; or is it because people regarded as attractive by society actually have valuable characteristics because of the way they’ve been socialised, having, for example, higher levels of self-confidence?

It’s hard to be definitive, but a Swedish study that looked at how engineering students were graded, first during real-world teaching, then in online teaching during the pandemic, found that there was a modest but statistically significant beauty bonus for students of both sexes before the pandemic, but that attractive female students lost 80% of this bonus after teaching moved online (it persisted for attractive male students). This certainly suggests a cognitive bias towards attractiveness.

The second big question is how much of our view of what makes for a desirable appearance is innate. That our view of what is beautiful is in large part culturally determined seems obvious: centuries ago, being fat was desirable; today, it is being slim.

Prejudices about sex and race play into appearance-based perceptions – is it any surprise that more masculine faces are associated with competence in political leaders, given the harmful sex stereotypes reinforced on children from the get-go? But at least some of what we see as attractive appears to be there from birth: a number of studies have shown that babies prefer faces ranked as attractive by adults.

Last comes the fraught question of how much we should be looking to unpick what could be described as “lookism”. It’s easiest when it comes to appearance bias associated with those characteristics against which it is plainly immoral and unlawful to discriminate, like sex and race. Making snap judgments based on someone’s genetic inheritance in relation to societal norms of beauty also feels wrong, and the same goes for weight, given there’s a genetic component to this too. But is it really unreasonable to react to how someone dresses, or how they take care of themselves, given how much many of us value self-presentation as an expression of our personality?

This is the aspect of appearance bias I find trickiest. I don’t want to live in a drab world where everyone dresses in grey and makeup is regarded as frivolous. As humans, we like aesthetically pleasing things. There is a joy in beautifying yourself through the way you dress, cut your hair, or wear makeup.

But not everyone enjoys it, and there is far more sexual objectification of women’s appearances. Social media influencers project unhealthy and unattainable standards of beauty to both girls and boys; research suggests that spending just seven minutes browsing Instagram feeds led to a decline in body satisfaction in girls, and body image issues are on the increase among young men too.

It feels like we live in a culture in which the emphasis on appearance is growing, and beauty norms are more and more unrealistic, determined by filters and tools that allow teenagers to do the kind of photo enhancement once the preserve of fashion magazines, and by an ever-lengthening list of cosmetic procedures and enhancements.

There is surely an aesthetic sweet spot. But it is precisely because so much of our appearances aren’t innate, but within our gift, that the beauty arms race is hard to call a halt on. There’s a hypocritical element to it all: I want the visible signs of female ageing to put woman at no less of a disadvantage than men, yet I won’t stop dyeing out my greys because I hate the way they look. The incentives not just to comply with societal expectations, but to actively enjoy doing so, are powerful, which makes that sweet spot very hard to find.

• Sonia Sodha is an Observer columnist

• Do you have an opinion on the issues raised in this article? If you would like to submit a letter of up to 250 words to be considered for publication, email it to us at observer.letters@observer.co.uk