The other morning, I sat down to watch The Nightmare, a new documentary about a condition called sleep paralysis. I was feeling tired, as so often over the past 35 years, because my night’s sleep had been punctuated and, ultimately, ruined, by the very same terrors the film depicts.

Sleep disorders are a common part of the human experience. From narcolepsy to insomnia, restless legs syndrome to sleep apnea, many of us find our nights are tormented, and our days ruined as a result. But few disorders can be as routinely terrifying as sleep paralysis. I had my first experience of it at around five or six. Just as I reached the point of unconsciousness, I’d wake, aware of where I was, but unable to move.

But it’s the hallucinatory aspect of sleep paralysis that is its most petrifying feature. As a boy, I would experience a frightening sound, somewhere between white noise and insect buzzing, while feeling a dark presence in the room. Sometimes I’d feel that I was leaving my body, but my journey out beyond my bedroom door was accompanied by sickening Hitchcockian lurches of perspective. Most commonly – and The Nightmare re-enacts this surprisingly common experience in a variety of disturbing ways – I’d feel that there was someone in the room, a dark presence slowly edging towards my prostrate body. Occasionally it would get to me, and would squat on my chest. I would fight to wake up, concentrating all my energy on getting one finger to twitch, as that would release my body from its prison. Eventually, that release would come and I’d sit up in bed, sweating, hyperventilating, the “shadow man” nowhere to be seen.

Many people with sleep paralysis suffer in silence. Around 30% of people questioned in a recent study reported experiencing it at least once, though a more conservative estimate suggests around 10% of the population get it frequently. Some worry it is a sign of mental illness, while others rationalise it in a variety of ways: one may see it as part of brain activity, deriving from the same complex part of our subconscious as dreams. Another may see it as an actual supernatural experience. The Nightmare found eight people to share their own experiences, from Manchester to New York. There’s Connie, a former atheist who found religion when she drove away her unwanted sleep visitors by invoking the name of Jesus, and Forrest, who made Halloween masks of the faces he saw.

Director Rodney Ascher found his subjects online, and, indeed, the early days of the internet was, for many, the first time people became aware that their sleep disorder had a name, and was shared by people around the world. YouTube is home to many videos of people describing what they have gone through in the night, from the banal to the chilling. But people have been describing sleep paralysis for hundreds of years, in different ways.

Google “sleep paralysis” and it won’t be long until you see Henry Fuseli’s 1781 oil painting The Nightmare – there are many who view this painting of a foul creature sitting on the chest of a sleeping woman as a perfect distillation of the sleep-paralysis experience. Similar depictions of beasts and hags are found through the history of art and literature, with nocturnal demons having variations in cultures throughout the world: the German mare is a folkloric demon that sits on sleeping people (Fuseli’s painting shows a mare watching the scene), the Turks have the karabasan, South Africans the tokolosh.

The Hmong people have a “pressure demon” that they call tsog tsuam. Bizarrely, 117 Hmong refugees from Laos who had relocated to America died unexpectedly in their sleep in the late 1970s, and perplexed researchers posited the existence of Sunds – sudden unexpected nocturnal death syndrome. The displaced Laotians, without access to their support networks and the shamans and rituals they usually employed to drive tsog tsuam away, were under considerably more physiological stress – perhaps what triggered the cardiac arrhythmia that killed them. Director Wes Craven read an article about the Hmong and was inspired to create Freddy Krueger, the ultimate haunter of sleep.

So vivid are the hypnagogia – the visual and auditory hallucinations that make sleep paralysis so upsetting for the sufferer – that some sleep researchers feel it is the most likely explanation for claims of alien abduction, and even cases of demonic possession. For me, it sometimes has a double kick. I’ll fight to regain consciousness and, having told myself I have done so, will still find that there’s some foul presence in my bedroom which then proceeds to punch me in the stomach. At this stage, my mind, which seconds ago knew it was experiencing sleep paralysis, is now convinced I’m the victim of a real-world demonic attack. And then I wake up.

Ascher decided against calling on doctors or sleep specialists for The Nightmare. Instead, he opts for a more immersive, experiential tone, an attempt to capture what it’s like for those who live with it, instead of describing it clinically. It pays off with some truly frightening set pieces. What might appear hokey in a horror film takes on a different timbre when it’s something you have gone through at 2am for more than three decades. There’s a spectrum of sleep paralysis; some people find their lives blighted by it and and fight to stave it off, others find it such a rich, otherworldly experience that they can’t wait for the next time it happens. My personal feelings are perhaps closest to those of Chris C, the person in The Nightmare who seems the most traumatised by what he has gone through. He, too, finds little solace in sleep, a time when, as he so perfectly puts it: “All the darkness looks alive.”

• The Nightmare is in cinemas now and available on DVD and demand from 26 October.