Life, as you may have had cause to remind your three-year-old recently, isn't fair. This is one of those truths that, beyond toddlerhood, we all like to think we understand. Furthermore, most of us agree that making life fairer would be a good thing, which is why politicians reach for the term "fairness" in even the most questionable contexts. (Nick Clegg, for one, appears to be conducting a vigorous extramarital affair with the word.) Thanks to Occupy Wall Street, and the protests it has spawned, fairness has rarely been such big news. But the Occupy protesters may face an even tougher struggle than they realise, because deep down, and whatever our political opinions, many of us seem to believe that life is fair – that by and large people get what they deserve.

This annoying but by now well-substantiated finding is known as the "just world hypothesis", and the most famous demonstration of it was a series of clever experiments by the psychologist Melvin Lerner. In one, he showed people what appeared to be live footage of a woman receiving painful electric shocks for making errors in a memory test. (She was actually his accomplice.) Some groups of viewers had the option of ending her ordeal; others didn't. The latter – forced to watch suffering with no chance of relieving it – formed far lower opinions of the woman, seemingly to "bring about a more appropriate fit between her fate and her character". Those opinions were worst when they were told the woman got no financial reward for her pains. The greater the injustice, the more people appeared to need to believe the victim brought it on herself.

What's going on here, it's suggested, is a quest for a feeling of security. The suggestion that victims of rape were "asking for it" is a case in point: if you can convince yourself that victims deserve to be victimised, you don't need to fear that you and yours – who don't deserve it, and would never ask for it – might have to endure the same fate. In a sense, it's the opposite of the "Ben Franklin effect", mentioned here before, which states that if you want to get someone to like you, you should ask him or her to do you a favour: to eliminate cognitive dissonance, the favour-doer will come to think of the favour-receiver as likable, since likable people are the kind for whom he or she does favours. The just world hypothesis sees suffering and concludes that people who suffer must be the kind of people we disdain.

In nastier corners of the positive-thinking world, this bias is explicit: victims of crime and even of genocides, certain dodgy gurus have said, must have caused their own victimhood. That's easily rejected. The more troubling thought is that the bias might influence us unawares: while consciously feeling bad for the less fortunate, are you subconsciously seeking to rationalise their fate? Maybe this also explains some people's tendency to blame themselves for all that befalls them, even though that redoubles their misery. At least "I'm a terrible person" makes for a coherent explanation; "I'm a good person, but life is randomly cruel" is much scarier. But it's truer. To quote that famously perceptive life-coach Clint Eastwood, as William Munny in Unforgiven, "Deserve's got nothing to do with it."

• oliver.burkeman@theguardian.com; twitter.com/oliverburkeman