What's the best question to ask on a first date? Many forests have been felled for books dissecting this question, but a few years back the dating website OKCupid.com made a startling intervention. If you want to know the chances of a first encounter ending in sex, its analysts said, the question to ask is, "Do you like the taste of beer?" If you're concerned with long-term compatibility, the question is, "Do you like horror movies?" When registering, OKCupid users answer hundreds of seemingly innocuous queries – the one about beer, it transpires, is most strongly correlated with positive views of first-date sex. Liking or disliking horror films, meanwhile, is a better predictor of a lasting relationship than beliefs about God. We could invent theories to explain this: maybe people who score highly on "sensation-seeking", as psychologists call it, enjoy both casual sex and cheap ways to get drunk? But the point about Big Data is that you don't need theories. The correlations are there, whether or not they seem to make sense.

But how many of us would embark on a serious relationship based on a shared passion for The Shining? When it comes to judging character, we prefer to believe gut instinct beats box-ticking. "We have a deep-seated need to feel that we can judge character," Jason Dana, of Yale University, told the Boston Globe recently. But many studies suggest we can't – and a new paper co-authored by Dana is especially damning. He looked at "unstructured interviews", the free-wheeling kind of interview that's usually crucial to getting a job. Participants were asked to predict the academic performance of two college students: in one case, they were given data on the student's past achievements, age etc; in the other, they got the data, plus a chance to interview the student. Consistently, interviewing led to less accurate predictions. The students who looked best "on paper" really were. Of course, you might have other reasons for wanting to see how someone handles themselves in conversation, but for forecasting performance, gut feel got in the way.

Technically, the problem with unstructured interviews (or dates) isn't that they're insufficiently informative. It's that they're too informative. Bombarded by data, we seek refuge in "sensemaking", clinging to stories that seem to render things clear. But those stories might include racist or sexist stereotypes about who's good at what. Or they might be the seductive stories of candidates skilled at interviews, yet rubbish at the job itself. "Because of sensemaking," the researchers write, "interviewers are likely to feel they are getting useful information from unstructured interviews, even when they are useless." Settling on a coherent story feels good, but that doesn't mean it's accurate.

This gap – between what our guts say and what the data says – will only grow wider. As Big Data quantifies more of our lives, we'll increasingly face dilemmas: if your instincts tell you to date or hire Person A, but the metrics point to Person B, whom will you choose? The economist Tyler Cowen argues that the future belongs to people willing to listen to computers: "Those who won't listen, or who rebel out of spite, will be missing out on glittering prizes." Whether this strikes you as liberating or chilling may say much about your personality. Then again, perhaps not as much as whether you fancy a pint.

oliver.burkeman@theguardian.com; twitter.com/oliverburkeman