Paul Chan has tried hot tea, hot baths, hot-water bottles, a cold breeze from an open window, mental maths, brainteasers, very slow breathing in bed and very brisk walks around his bedroom. Now, on a random night in October, the 52-year-old from Liverpool tries to get to sleep by imagining that he is James Bond. Why not? Chan is among that enormous proportion of the British public – one in three, according to an NHS estimate – who suffer from routine bouts of sleeplessness. He has just been to the cinema to see No Time to Die and as he closes his eyes for the night, he decides to start at the beginning, mentally recreating the movie in as much detail as he can manage. Scene one. A frozen lake in Norway …

A little way north, in Durham, Lucy Adlington is alert, awake, and stuck. The 36-year-old silversmith cannot fall asleep but is hesitant to clamber out of bed for fear of waking the rest of her household. Somewhere between 3am and 4am, she picks up her smartphone and, speaking softly, begins to dictate a voice message. What does it feel like, being awake, alone, out of options, in the smallest and quietest hours of the night? “Like being an animal in a cage,” Adlington says, murmuring into her phone.

She is documenting her experience of a wakeful night because, a few months ago, I went in search of insomniacs. In particular, those who would be willing to further interrupt their nights to describe the sensations and frustrations of insomnia, helping to paint a picture of sleeplessness in Britain in 2021, one night-time voice note at a time. Some of those who respond to my request on the Guardian website are close to despair. Many are seeking treatment, trying tricks, open to wheezes, superstitions, suggestions. Quite a few have come to shrug and accept their condition. People find their insomnia stressful, ridiculous, useful, cruel. As Adlington can testify, one of the worst aspects of the condition is its self-sustaining nature. “So frustrating,” she tells me, “because anxiety causes the insomnia. But when you know you’re going to feel tired and terrible tomorrow, the insomnia causes the anxiety.”

In Essex, Freddie Lewis, a 17-year-old college student, has been using his hours awake to do some homework. “Trigonometry,” he croaks, in a 2.40am voice note. Lewis was convinced that some difficult maths would help lull him to sleep. “Didn’t work,” he concedes, finally wandering downstairs to the family lounge to watch TV.

In Manchester, marketing agent Joe Harper, 31, is refusing to abandon his bed just yet. “It’s 3.25am. I haven’t slept at all tonight,” he says, running through the list of questions he’s been lying awake asking himself.

“Have I drunk too much caffeine? Have I spent too much time on my phone? What have I eaten? Am I drinking enough water? What am I worried about? What do I want to do at work? What do I want to do in my personal life? What do I want to buy my girlfriend for her birthday?” To counter this endless, nagging scroll of doubts, night after night, Harper has tried reading for hours in bed. He has tried herbal remedies. He has tried doing more sport during the day. He has tried doing less sport during the day.

Every long-term insomniac with the ability to Google knows the listable reasons they may not be sleeping well. Anxiety. Depression. Guilt. Regrets. Late meals. Too much booze. Caffeine. Irregular bedtimes. Noise from the street. Porous curtains. Crap mattresses. Side-effects from prescription medication (or from recreational drugs). Overactive glands. Undiagnosed organ conditions. The menopause. The fear of tomorrow. The fear of death. As for possible cures, “I have tried literally hundreds of online and home remedies,” says a property lawyer in Bath, an insomniac for 15 years. “I have read just about every article ever written,” says a retired lecturer in Bolton, an insomniac for 65 years.

As Katy Cowans, 48, a lawyer from Barnsley, puts it: “Us veterans? We’ve heard of every cure, trust me.” Try earmuffs, they’ve been told, or eye masks. Warmer milk or a cooler bedroom. Try meditation, mindfulness, acupuncture, or something from the therapy bucket (cognitive behavioural therapy, stimulus control therapy, sleep restriction therapy, relaxation therapy). Bin your bedside clock! Deactivate your phone! Don’t catnap! Don’t smoke! Most of the insomniacs I speak to have tried sleeping pills at one time or another.

There’s an A to Z of these pills: antihistamines and amitriptyline to zopiclones and zolpidems. Adlington in Durham has been prescribed some antihistamines by her GP, not that they are doing much for her tonight. Renny Whitehead, 47, a photographer from Brighton, used to take the stronger zolpidems, but that was when he was working topsy-turvy hours as a flight attendant. Tonight, he pops a milder zopiclone. While he waits for the drug to kick in, he records a voice note.

Whitehead can’t actually remember responding to my request for insomniac volunteers, back in the summer. His sleep was terrible at the time, he says. His brain was mush. He figures he must have volunteered to take part during the middle of another restless night, forgetting about it by the morning. Tonight, when the zopiclone kicks in, Whitehead sleeps well, for about an hour. Then, abruptly, he wakes. It is about 1am. He tosses and turns.

Around sleepless Britain, as pills are being swallowed, alternatives to medication are being trialled. A man in Warrington says he finds the BBC World Service helpful; he listens through a special under-the-pillow speaker. A grandmother in London reports that switching from the World Service to Radio 4 meant she avoided regular overnight news bulletins that tickled her curiosity and nudged her out of unconsciousness. Digby Cox, a retired civil servant in Derbyshire, has started writing long emails at night. “My family tell me they always know when I’m not sleeping,” Cox says, “because their inboxes fill up.”

Fabio Sorbello, 44, from Cheltenham, uses an array of methods to get to sleep. Press-ups. Classical guitar practice. A microdose of alcohol, “less than a shot”. Meanwhile, the property lawyer from Bath prefers to stage imaginary football matches in his head. “Second by second in real time, possession-based tiki-taka, no goals.”

The retired lecturer in Bolton favours novels, but only bland ones, she says. Hilary Yallop, 32, a doctor based in Sheffield and the mother of a 15-month-old baby, uses her time awake to catch up with a friend living in Hong Kong. Otherwise she tries to “complete” some item from the news, Yallop says, reading every article on it she can find; or counts the dots on the wall of her daughter’s room, “like counting sheep … Sleep deprivation does strange things.”

Chan, meanwhile, is still out on Her Majesty’s Secret Service. By now, the Liverpudlian has made his way through a painstaking recreation of the first half of the Bond movie. He has choreographed fights. He has travelled between glam locations. But he isn’t asleep yet, and picks up his phone to record another groggy voice note. “I think I must have done 40-odd scenes,” he sighs. Will oblivion ever come?

***

Chan’s trouble with insomnia started when he was a boy, living above his parents’ takeaway restaurant. “It never got quiet before midnight,” he says. For Adlington, the sleeplessness began about six years ago, with night terrors that jolted her awake and set her up for bad nights. It was the Covid lockdowns of 2020 that knocked Lewis out of his usual sleep routines, and he hasn’t found a way back yet. A lot of mothers are experiencing worse sleep due to pandemic-based anxieties, according to a 2020 study by the Centre for Population Change at Southampton University. The same researchers also found that sleeplessness among British people of colour rose steeply last year, again attributed to Covid worries.

Cox, the retiree in Derbyshire, recalls that he rarely slept for more than four hours a night during the years he was caring for his wife, Jenny, in the final stage of her life. When Jenny died in March 2020, Cox slept comparatively well for a time, he says. Then something changed. He started going for weeks at a time on what felt like only an hour a night. “When I’m on this insomnia thing, it’s like there’s a wall I can’t penetrate,” Cox says. “All night, all my senses are alive.”

Cowans, the Barnsley lawyer, is certain that her own trouble sleeping stems from traumatic experiences in her childhood. Cowans had lost two siblings to a genetic illness by the time she was 14. Before they died, she became accustomed to exhausting overnight vigils in the local hospital, “listening to Radio 1 with them until it stopped at 2am. Not much sleep, then straight off to school. Doesn’t take a specialist to figure out where the insomnia might have come from, right?” In her 20s Cowans was a music journalist, up all hours at gigs. “The insomnia actually came in handy.” However, she retrained as a lawyer and now has to be at her desk by 9am, come what may. She has two daughters who need to get off to school around 7am.

On this night in October, her insomnia no longer any use, Cowans tries reading a thriller in bed. It gets later and later. Soon she is counting the hours remaining until her early start. This insomniac’s algebra – an incessant and despairing totting up of the maximum possible amount of sleep left to be squeezed from a night – is described in one way or another by many people. Cowans has five hours left. After her daughters are off to school, she figures, she might be able to sneak in a 30-minute doze at her desk before she logs in for her first Zoom meeting of the day.

For many of the respondents who contribute to this story, the boundary between employment and rest has blurred. A 56-year-old public sector worker from Devon does housework at 3am. A 50-year-old tutor from Birmingham does the same, dabbling in a bit of poetry writing, too. “The best ideas come at night,” she reports. Sometimes, Chan finds himself designing bizarre items of furniture in his head (“A bookcase that also functions as a ladder? Made only of plywood?”). Cox has been known to garden in the dark. He once cut down a tree before sunrise.

Zoe Jewell, 35, a London film-maker whose son is only a few weeks old, is among the hundreds of thousands of parents around the country who are up and awake in the night, feeding, changing nappies. So far, Jewell reports, “I’ve found being up in the small hours a strangely peaceful experience.”

In Durham, Adlington has been trying to heed the advice of the philosopher Alan Watts, who believed that if you can’t sleep, you ought to make productive use of the time. At 4.20am, the silversmith takes the risk of rousing her household, clambers out of bed and opens her laptop to work. Forty minutes later, at 5am, Whitehead’s decision about sleeping or working is taken out of his hands. His alarm clock buzzes, letting him know it is time to get up and drive from Brighton to London for a photography job. Reluctantly, Whitehead begins his day.

He records another voice note from the road, comparing his own disjointed night with that of his wife, who slept beside him. She seems able to approach bedtime with something like relish, he says, while for Whitehead there is only dread. “That lovely being-surrounded-by-pillows-and-blankets thing, I just don’t get that any more.” Cowans’ partner, who lives in another household, sometimes texts her in the evening, not to say “goodnight” but to say “good sleep”. Sorbello, in Cheltenham, says he is sleeping about as well as he has in 20 years, a fact that he puts down in large part to having found a new partner.

***

The somnolent other halves of insomniacs play a funny role in all of this. I know because I am one. On my side of the bed, most nights, I can rely on regular blackouts. My wife is not so lucky. I sometimes become aware of a bedside lamp turning on. Maybe she’s reading. Maybe she has her big headphones on to listen to music. If I feel anything, through many layers of unconsciousness, it’s partly pity (that she should be suffering) and partly relief (that I can roll over and carry on with my zeds). In the morning, more than anything else, there’s guilt – that I should have enjoyed a night’s sleep that could not be shared.

There are insomniacs who are, frankly, and fairly, jealous of their sleepier bedmates. A retired Samaritans worker in Leeds explains that her husband “can sleep on a brush”, which prompts long nights of envy that leave her feeling ashamed. A charity worker in West Yorkshire says she feels notably uncharitable towards her partner’s snoring, “the sound boring into my brain”. Sometimes insomniacs pity us sleepyheads, though. In 2019, The School of Life, a mindfulness organisation based in London, published a book that set out to exalt the benefits of insomnia, “so we may feel less persecuted by, and alone with, our sleepless nights”, as the anonymous authors wrote.

Strange insomniac advantages were described in the book. What a chance, wrote the authors, to make all those excellent, imaginary speeches that eluded us during the day. What a chance to observe a loved one without their guard up. “We can see again the person we first got together with. We can focus on the details we found so endearing … ”

One night last summer, awake again and fiercely missing his wife, Cox went out to the garden and grabbed some paint from his shed. He took himself off for a walk, eventually making it to an underpass near his home. What followed was out of character, Cox tells me. “I’m a retired civil servant. Graffiti would have been unthinkable for me, once. But with sleeplessness comes a different perspective. And I just had the urge to do something.” He found an empty piece of wall, and sprayed a heart.

***

Chan has slept! A bit! Thinking back over his night, the next morning, he realises he got as far as Bond’s journey to the bad guy’s island before he conked out. Adlington, up at 4.20am, spends the whole of the next day feeling like a phone on 20% battery. “It’s something you end up living with,” she concludes. “And so you learn to live with it.” Lewis was still awake in front of the telly when his father woke up the next morning (Dad made them both a consoling breakfast). Cowans gets her kids to school, but then feels too guilty and conscientious to sneak that catnap at her desk.

“When you don’t sleep well, you worry about it, and when you do sleep well, you overanalyse it,” says Harper, who passed a mixed night of his own. Harper says he has reached a sort of truce with his insomnia, learning to “kind of give it a nod of acknowledgment” and laying down his weapons against it. “The hours can sometimes feel like they drag by. But they can sometimes feel like they fly by, to be honest, and you can end up having some really interesting thoughts, some really clear views on life.” For Harper, a turning point came when he realised that, however often he felt like the only person in the land still awake, he cannot have been. Going by the NHS estimate, there would be many millions of others in Britain, on any given night, with their eyes wide open like lizards.

A lot of respondents mention this – the solitude of insomnia, and their sense of being abandoned by the easy sleepers of the world. In her own tired murmurings, Cowans started to wonder about her neighbours. “So quiet,” she muttered in a 2am voice note. “I wonder if anyone else on my street is awake … I’d love to know.” Insomnia is that most contrary of afflictions, common as the common cold and a near universal human experience, yet one that leaves people feeling utterly alone. “But you’re not alone,” Harper says. A 31-year-old teacher in Manchester concurs. She reports the curative effects of reminding herself, every so often, that there are different time zones around the globe and at any time, fully half of the human population are wide awake, too. “Stopped me feeling quite so forsaken,” she says.

If nothing else, there is always the promise of tomorrow night and that elusive perfect sleep. Cox has been through all sorts of eccentric nights in his time as an insomniac. Nights when he chopped down trees. Nights when he went out on impromptu memorial missions, paint pot in hand. Nights, he tells me, when he walked around his deserted village at 4am, headphones blasting Brahms and his arms conducting wildly, there being nobody around to see and snigger. This night in October is different. While a silversmith worked on her laptop at 4am, and a sixth-former watched TV, Cox slept through until morning. Eight hours. Bliss.