It's about 3.20am. I know this not because I've checked the clock recently – I just know it's about 3.20am. Let's add it up: went to bed at 10.30pm, read for a bit, lights out, turned over a few dozen times (midnight), moved upstairs to avoid waking my wife, meditated for half an hour (12.45am), tried again, turning and turning (1.45am), went to look at the sleeping children, turned the pillows over to find the cool side (2.15am), put pyjamas on (2.45am), turned over on to my front to try and squash myself to sleep, tried sitting up, kneeling, and – in a moment of absurdity – tried going to sleep standing up. So that would make it past 3am.

I fight the urge to check the clock. It always says terrible things like 03.38, or 04.29 and never 22.17. Checking the clock just amplifies the anguish: that I am awake, have been for hours and will be, quite possibly, for ever. What a fictional horror that would make: someone who has forgotten how to sleep.

For now at least, that someone is me. My rational self – which has been having quite a tough time of it lately – knows that this will pass. In a while, a few days perhaps, maybe a grotty week or two, I will reemerge on the other side and ask myself what all the fuss was about. Then, I will approach my bed and my pillow with a shrug, climb in and go out like a light. You go to bed, you sleep. It's easy, right? I'll enjoy several weeks, months even, of perfectly average, unremarkable sleep. And then suddenly, like flicking a diabolical thought-wave machine, it will start up again.

Until recently, I didn't struggle too much with sleep. There was a strange week once. It would have been 1982, because the Falklands war was going on and I think I was quite troubled by it. I was 12 and starting for the first time to feel a bit unsure about things. I lay in bed, night after night, and just couldn't sleep until 1 or 2am. I would wake my parents, and my dad would come in and explain that it didn't really matter. Lots of people, he said, had to stay up all night to do their job. He used to be in the navy, and so used the example of a duty officer on deck, in command of a frigate or a destroyer on the high seas through the night watch. I thought of ships sinking in the south Atlantic, and became more wakeful than ever.

Apart from that week in 1982, sleep was a neutral presence in my life. I have calculated that I have had 20 different homes since I left the house I was born in, 24 years ago. There have been a wide variety of beds in that time: student mattresses on creaky floorboards, Soviet sofa beds with envelope sheets into which you folded a blanket; a nasty specimen supplied by a cheap landlord that felt like it had been fashioned from a box of rusty spanners tied up with string. I slept like the dead in all of them. But after I had a nervous breakdown in 2009, sleep was one of the first things to go. As I've got better it has been one of the last things to resolve itself.

The three phases of night

For the insomniac, different times of the night have different qualities. The time between 10pm and 11.30pm is the golden age that you want to hold and cherish and keep forever. This is the time that you feel nostalgic about when the clock reads 04.29. For at this time, there is still a chance that you will disappear suddenly into the void and, the next thing you know, there will be birdsong and milkmen and dawn tiptoeing into the edges of the room. Still a hope, still a chance.

The second phase is unsettling and takes you up until the streets are quiet and the house is cold. Here the feeling is that if you do manage to get off to sleep, all is not lost. The next day will be OK; you will have cause for mild optimism in the morning. And the longer you are an insomniac, the more you recognise that this is your real window. The mind and body, no matter how ravaged, still need sleep, and they'll take it – if you let them.

As I've got better at this, I would say that as much as 50% of the time I get to sleep during phase two. But it's difficult. Loneliness and restlessness edge in. It's sad too. These hours of the night used to mean something so different: fat basslines and firm friendships – the nights when everything was funny and lack of sleep was a badge of honour. Now it's different. Now I know what Thelonius Monk meant when he wrote "It begins to tell 'round midnight".

After 2am things get wretched. This is when, if you've been doing this night after night, you start to ask those unanswerable questions: why me? What did I do to deserve this? What is happening to me? These lines of thinking are catastrophic and totally disproportionate to what is actually going on. It's just a bit of sleep, that's all.

And yet, in the small hours (such a misnomer: they rarely feel small), the tendency is to feel like it's the end of the world. By now, pretty much everyone across the continent is asleep. The bed is in almost as bad a state as you. In theory, you're supposed to get up and read. In practice, you can barely finish a sentence without the worry blurring the words on the page.

A passive pursuit

I should say at this point that for me, it hasn't been the wakefulness of the busy executive with his deadlines and to-do lists, or the furious restlessness of the self-employed churning over how to get tomorrow's money today. This affliction is all in the passive mood. It all happens to you, like some capricious reflexive verb that has yet to be invented.

In Russian, they don't say "I can't sleep." They say: "It isn't sleeping to me." I like that. For a long time now, it hasn't been sleeping to me. Not properly anyway. From 2am to 4am you will really fight the thing, like it's actually in the room with you. That's exhausting. After 4am things could get truly horrible, but by then you're probably so bereft that you'll drift off at some point into thin dreams about empty cities and lost causes. But not for long. By now, you're desperate for morning so you can move on, write off the night, hope for better luck next time.

A few nights of this is bewildering. Your bones ache like cold plates of uranium. You can feel the shape of your skeleton. You have startling moments of sudden realisation: this is me, I am in deep trouble; this is a vicious circle, it won't be resolved.

And there is, seemingly, nothing that you can actively do about it. Falling asleep is one of those perverse things that are harder to do the more you try. In general we have become conditioned to thinking that the more effort we put in, the more we will get out. That's clearly not the case with sleep. Trying to drift off just doesn't work. Sleep is a reflex and needs to happen effortlessly. When it stops happening and you start trying, problems begin.

I've tried everything. Some things do help. First there are pills. These are safe and do a job, but are not viable as a long-term solution. That way, one problem can easily become two. For a week, it's fine to get over a short, sharp spell. But insomnia tends to be a long-distance race. You need something else after the first mile or two.

There are small changes you can make to your routine. These include the following: only use the bed for sleep; don't go to bed until you are sleepy; limit television and exciting books before bedtime; try a hot bath or some hot milk; be vigorous during the day – but not in the evening; make the bedroom dark and cool; get up if you don't go to sleep within 20 minutes; get up at the same time in the morning (don't lie in); avoid alcohol, caffeine and cigarettes in the second half of the day.

Three ideas for insomniacs

There are three other things I would add from my own experience of insomnia. Meditation helps in two ways. Firstly, a half-hour session in the middle of the evening can really slow the heart rate and make you feel sedate. I'm told that after a few months, meditation can actually induce neurological change. Secondly, when you're lying there wondering how to slip off, some breathing exercises, in which you fill your mind with observation of the breath coming and going, can usefully divert the restless mind.

Stop thinking. It's really hard to do. But if you can just stop the mind from broadcasting its peculiar mix of propaganda, impression, recollection, projection, apprehension and sedition, you invite in a nothingness that is a close cousin of sleep.

Acceptance. The hardest, but most valuable lesson of all. Essentially it boils down to this: bad things happen to us, but they are nothing compared to the bad things we do to ourselves. We make things worse by our constant value judgments ("I should be asleep"), our ceaseless comparison of ourselves to others ("my wife and kids are asleep, why aren't I?"), and our unrelenting standards for ourselves ("If I don't get eight hours' sleep, I'll be a wreck tomorrow"). Acceptance is about seeing those statements for what they are: unhelpful interventions in what is already a tricky situation.

We are who we are, and no amount of fretting – particularly at 1.30 in the morning – will help. What if instead we were able to say: "I'm not sleeping too well at the moment, but it won't hurt me. My body will sleep when it needs to again. I can always catch up." I can safely say that every time I lie awake agonising that if I don't sleep it means I'm not as well as I think I am, I will not sleep. Whereas if every night I lie and think: I am what I am and there is little I can do about it, sleep comes more easily.

Depression - the illness that's still taboo