Perhaps I should feel more concerned about my wife's habit of apologising for me before I meet anyone she knows. The truth is, I'm not even sure what she's apologising for, except that I'm occasionally not that chatty. And I fidget. And my eyes stray about the place when people are talking to me. And I sometimes ask questions that can come off as a bit direct. There was that time, too, at the engagement picnic in Hyde Park, when I excused myself from all the socialising and went and stood by a bush. She was cross about that. I was sorry, but I didn't think anyone would notice. All that chitter-chatter felt like having my head squeezed.

So although I should feel worse, I don't, because it means the occasions on which my wife invites me out are becoming ever rarer. Over the last few years, I've come not really to like out. I work alone, hike alone, go to the cinema alone, eat at restaurants alone. Once a year, I even holiday alone. As soon as possible, I intend to move even deeper into the countryside. The reason is people. I used to like them. Then something happened. And now I don't.

I am solitary by nature, and solitude isn't a vice. It isn't binge-eating junk food or abusing drugs and alcohol, so I've always felt able to indulge myself freely in the soft joys of nobody. But then I started hearing that, health-wise, it might be dangerous; that you can overdose on alone.

I'm reading a book about why this might be when my dog starts barking. I peer out of the window. A man is stealing my sandbags. The village in which I live has flooded and those sandbags were hard won. I grumbled to the council, then complained to a man in a lorry filled with sandbags who told me he wasn't authorised to give me any. It was only when my wife asked him that, for some reason, he changed his mind. And just in time. As I step out in my slippers to shout at the man, the water is only two feet from our door.

"Oi!" I cry.

"You don't need them just yet," he says, "but around the corner it's urgent."

"You could've bloody asked," I say.

"I didn't think anyone was in," he says.

"You didn't even knock!"

"I didn't," he agrees.

"Are you going to bring those back?"

He shakes his head in anger, says "Yes!" and disappears around the corner.

People! You see what they're like? Anyway, where was I? Loneliness, by John Cacioppo and William Patrick (WW Norton, £12.99), page 14,"which makes each of us, to some extent, the architect of our own social world. When loneliness takes hold, the ways we see ourselves and others, along with the kinds of responses we expect from others, are heavily influenced by both our feelings of unhappiness and threat." I wonder, can it be true? That the unhappily friendless create their own state of isolation? But surely you can't be described as "lonely" if, like me, you're alone by choice?

According to the book, our particular level of need for social inclusion is inherited. Some of us don't need so many friends. The pathologically lonely, though, sound as if they can be difficult. They tend to imagine people are "more critical, competitive, denigrating or otherwise unwelcoming" than they really are. "Fear of attack fosters a greater tendency to pre-emptively blame others." This fear can also make them lash out, become desperate to please or cause them to play the victim. Those poor people.

The dog's barking again. I'm hoping it's the thief returning my sandbags, but it's an annoying, smiling man with some leaflets about flooding. "I'm from the Environment Agency," he says. "We're just going round checking everyone's OK."

I squint at him suspiciously. "OK. Well, I'm fine."

"Great," he beams. "There's such a wonderful atmosphere in the village, isn't there? Everyone's out, helping each other."

"Huh!" I say. "Someone's stolen my sandbags."

His face falls. I have an unexpected onrush of something that feels like shame. After I close the door, I recognise the moment as the kind my wife sometimes complains about. She'll tell me I've been rude, and I'll be mystified and panicked. Was I? Why? I was just… "It's like you live in another world," she'll reply.

I call Professor Cacioppo, co-author of the loneliness book. He's a neuroscientist who, 20 years ago, felt his colleagues were making a mistake by viewing the brain as a standalone organ. Because humans are a highly social species (one famous psychologist, Professor Jonathan Haidt, describes us as "part bee"), he theorised that our brains must be designed to function correctly only when they're connected to other brains. To test this idea, he studied brains that lack sufficient social connections. "That condition, of course, has a name," he tells me. "And it's loneliness."

Cacioppo's breakthrough came when he found that, when they sleep, the lonely suffer more "micro-awakenings" in the night. His point isn't merely that they typically feel more fatigued (which, incidentally, they do). For Cacioppo, this was evidence that they experience the world in an entirely different way. "Take any social species, such as fish," he says. "If you're on the perimeter, you're more likely to be predated. Your brain goes into self-preservation mode. You become more aggressive, more anxious, more depressed, there are changes in sleep. Why? Because it's dangerous. You show micro-awakenings because your brain remains partially alert for the presence of an attacker."

Not all scientists agree with Cacioppo. Appeals to evolutionary principles for explanations of behaviour – we show micro-awakenings because of a primal fear of being eaten – are sometimes rejected as speculative Just So Stories, because they're untestable. Also disputed is his conviction that it's not the number of friends that counts but how we feel about them. "It isn't objective isolation," he insists. "It's whether you feel isolated. The brain's not sitting there counting people."

I ask him to define "friend".

"It's to do with synergism," he says. "Let's say I have to move some furniture. If I'm doing it alone, I'm likely to injure my back. If two of us are doing it and we act as individuals, we'll both be likely to hurt our backs. But if we each take a side of the table, it's transformative. It's synergistic. We change the nature of the challenge."

I have two relationships that sound like that. The first is with my wife, who's mostly not here, because of work. The other is my friend Craig, who lives in Sydney. Cacioppo's book features a questionnaire, the UCLA loneliness test, which I took before we spoke. "How often do you feel outgoing and friendly? How often do you feel 'in tune' with people around you? How often do you feel that your relationships with others are not meaningful?" I scored 63.

"That's really high," Cacioppo says.

"It is?" I say.

"Yes," he says. "It is."

My journey into the quiet took perhaps 20 years. As a teenager, I'd constantly agitate my friends to meet outside Woolworths on a Saturday afternoon or go drinking stolen amaretto in the woods. When they'd sometimes say no, I'd be mystified. How could you possibly not want to go out? It was fun! It was drama! It was life! I had friends, but also plenty of enemies. On at least two occasions, I somehow managed to turn almost everyone I knew against me. I was loud, back then. Disruptive. When I left school, I found an older set of associates, all my classmates having fled for university. One of them once told me, "When everyone slags you off, I always stick up for you." I tried to get on with people, but seemed mostly to alienate them. It was confusing. How do you make friends? What do you do? It didn't help when I drank, and behaved as I behaved. And I drank a lot.

I got sober at 26. I started socialising alone and found it wonderful. Friday nights would be spent in my rented room with a DVD and some Doritos. I no longer struggled to get on with other people, because there weren't any under my quilt. As a writer, I came to appreciate the interview as conversation in its ideal form: I'd ask questions that would ordinarily be considered rude, while my subjects invariably asked nothing. When people say to my wife, "How can Will work alone all day in that dark room?" she tells them, "He loves it." And I do. It's safe in here, with the blinds pulled down. By writing, I get to talk, without the pressure of the listening face. My deepest intimacies are shared with the blank page on my computer screen. I confide in it things I keep from my own family. In a way, you're my closest friend.

But having almost no social connections triggers strange symptoms. Like, I'm drawn to public transport. The top deck of the bus is the perfect party: enveloped in the comfort of the crowd, yet safe in the knowledge that no one will speak to me (and I'll not be sorely judged for preferring not to speak to them). After days of not talking to anyone except my wife, I'll sometimes find myself unable to stop. An editor will phone and I'll pour words down the receiver, fast and burbling, only to be left with a hot combination of embarrassment and exhilaration when it's over. On the occasions I do socialise, and it goes OK, I'll feel so high that I struggle to sleep. I'm obsessed with reality TV. Contestants on Big Brother come to feel like friends. I care more about Imran on the Fried Chicken Shop than I do my own neighbour. Two decades after I left the drama of its corridors, I still dream about school.

"Over the years," I tell Cacioppo, "I've thought the problem is maybe that I'm just grumpy, or antisocial or depressed. I never considered loneliness."

"It's not that you aren't depressed or anti-social," he says. "Those are consequences of loneliness. You can feel very comforted by the fact you're normal."

This is loneliness's predatory irony. The more alone you are, the more others want to leave you alone. The more others want to leave you alone, the more alone you want to be. And so it goes, until you're there, with the blinds down, scowling at anyone who comes to the door. When your only contact with the human world is news reports of scandal and murder and the narcissists and witch-finders on Twitter, your sense of what people are actually like becomes distorted. You begin to fear them. When I'm not otherwise occupied, the individuals in my life rear out from the corners of my imagination, each a potential enemy. I have fantasy arguments in my head, compulsively rehearsing every possible fight I might have in the future. I even make the faces: angry, insulted, outraged. I'll be walking to the shops, clenching and re-clenching my fists, not realising what I'm doing until a passerby looks at me, alarmed.

The social world becomes a place of war, and everyone in it a villain. But it's a trap, this way of mind, it's a trick, a terrible illusion. When storytellers create characters that display the traits of the lonely, it's us who are the villains. In life, we're the tutters, the eye-rollers, the complainers; we're the ones who turn the comments sections toxic; the ones whose doorbells children dare each other to ring. I can guess what the sandbag thief and the leaflet man think of me – and, for that matter, all the others who live around here. I make a confession to Cacioppo. "Sometimes," I tell him, "I think the real problem is I'm just an arsehole."

"It's not a character thing," he says. "When something negative happens, and you're concerned about yourself, that's not because you're not a nice person. Your brain is in self-preservation mode. You're thinking about what that negative event means for your own survival. All brains do this, but it's bad to stay in that state."

Quite how bad comes as a shock. Trying to understand how our ideas about the world can affect our physical bodies is genomics researcher Steve Cole. He often describes the human body as "permeable", as if it somehow absorbs the events of our days. "People don't like this idea," he says, acknowledging that this is early science, and disputed by some. "But the more we look at it, this permeability thing is kind of inescapable."

In one small pilot study, Cole found loneliness can trigger inflammation, which is the body's way of helping immune cells reach infections and encouraging the healing of wounds. "Inflammation is the first line of defence against injury," he says. "It's as if the brain perceives the world as threatening and activates this defensive response before there are actually any microbes or injuries there. But this bubbling background inflammation is fertiliser for everything that kills us. It helps the development of atherosclerotic plaque, so you're going to have a heart attack; it helps disable brain cells, so you've got a neurodegenerative disease now; it helps a nascent cancer cell grow and metastasise." Cole's study also found a decline in the systems that defend against viruses. "Loneliness basically rivals cigarette smoking for its total association with mortality risk. So it's pretty big."

That evening, the man fails to return my sandbags. I wonder if he might have done had I responded to him differently. Worried about the flood, which is now just steps from my door, I walk around the corner to find them being used to corral a stream of water into a bubbling drain. Under the irritated gaze of the affected homeowner, I lug them back, one by one. Then I stop and return. With a smile and an apology, I explain who I am and why I need them. We have a chat. As it turns out, he's quite nice.

This is the paragraph in which I'm supposed to write how I'm going to change. After all, excess solitude has curdled my personality and my long-term health might be at risk. But it's not so easy. Loneliness is a passive compulsion; to binge, I need only do nothing. I have, however, recently made two social arrangements with new people. I don't know if they'll be a success. There's a good chance the occasions might end up being awkward or weird, and my wife might have to say sorry. All you need to do – my perfect, wordless friend – is be thankful you don't have to be there. 

• Will Storr's latest book, The Heretics: Adventures With The Enemies Of Science, is published by Picador at £8.99. To order a copy for £7.19, including free UK mainland p&p, call 0330 333 6846 or go to theguardian.com/bookshop.