Every day, Kyla* travels to a fictional universe with advanced space travel. It’s not real, of course – but an incredibly vivid daydream, centred on a protagonist with a detailed history. “It covers 79 years in the life of my main character,” she says. “I know how the whole thing plays out, and I can drop into it at whatever point I want to experience.”

Today, this habit is pure entertainment, which she limits to just an hour a day. “It’s like watching Netflix,” she says. “I just go into my head and enjoy it.” In the past, however, she had felt that her fantasies had become all-consuming. “There was a point where it was like an addiction.”

Karina Lopez tells a similar story. Her daydreams centre on conversations with different characters – some real, some imaginary. She’ll replay the same scenario, tweaking the details – a process she finds incredibly pleasurable. “As soon as I wake up, I want to daydream.”

At college, she would become so lost in these imaginings that she would forget to study for her exams or run errands. “I put off so many things – but in the moment it feels so good,” she says. On average, she now spends about three hours a day immersed in daydreams, but on bad days in the past, she could spend as many as six hours locked in her inner world.

Such reports are of increasing interest to psychologists, who have started to identify a subset of the population marked for their unusually immersive daydreams. At their best, these vivid and compulsive fantasies can be a source of pleasure and comfort, but they can also be a serious cause of procrastination and distraction, and can prevent people from maintaining their social connections, looking after their health or even eating regular meals.

With research revealing that as many as one in 40 people may experience these problems, it seems increasingly likely that “maladaptive daydreaming” will soon be formally recognised as a psychiatric disorder. So what is it? And how can it be treated?

***

Prof Eli Somer, a clinical psychologist at the University of Haifa in Israel, was the first to identify the phenomenon. In his practice, he came across six patients who described entering vivid fantasies as a way of soothing their psychological pain.

In the wake of a romantic breakup, one patient simply continued the relationship in his head; another, facing extreme loneliness, would imagine the conversations he wished he’d been able to have. “It’s an escape from what is happening in the here and now,” a third patient told him. “There are many circumstances in daily life that frighten me. Daydreaming helps me not feel the fear.”

Somer recognised their reports as a form of dissociation that had not been described in the scientific literature before, and so, coining the term maladaptive dreaming, he wrote a paper describing the phenomenon for the Journal of Contemporary Psychotherapy.

It was immediately apparent that these intense fantasies were very different from the kind of mind-wandering the average person might experience. “Mind-wandering can be fleeting thoughts,” explains Dr David Marcusson-Clavertz, a psychologist at Linnaeus University in Växjö, Sweden. “You might be reading a book and then spontaneously think of an old friend.” While the people with maladaptive daydreaming might also be prone to these distractions, their fantasies are complex, detailed and compulsive.

Consider the experiences of a maladaptive daydreamer called Michelle. Her daydreams have involved international travel, working as a reporter at a disaster zone and conducting important research about Covid. The story she constructs is often so complex that she can spend hours finding the specific details on the internet to fuel the fantasies. “In my head, I see it very clearly – as if I was picturing what I did yesterday.”

Many maladaptive daydreamers report being prompted by regular movements – and they may even use rocking motions or pacing to get into the correct mental zone, a little like self-hypnosis.

Despite the sheer detail of their fantasies, immersive daydreamers do not confuse their fantasies with reality, and they don’t tend to come out of nowhere. “It’s voluntary – it’s not intrusive,” says Somer. This makes it different from psychosis, in which someone has less awareness of their mental state, and the daydreaming is not – by itself – harmful for someone’s mental health.

The problems come when it is taken to excess. As Somer’s original paper had noted, many people use their daydreams to escape from negative emotions. This might offer short-term relief, but it can prevent the person from confronting the issues that may be at the source of their distress. Along these lines, a recent study by Somer and Dr Nirit Soffer-Dudek, of Ben-Gurion University of the Negev, asked participants to keep a daily record of their feelings and behaviours over a two-week period. They found that negative emotions often rose after a day of particularly excessive daydreaming.

A study by Prof Alessandro Musetti at the University of Parma in Italy, meanwhile, examined people’s reactions to the early stages of the Covid-19 pandemic. He found that maladaptive daydreamers were especially likely to experience higher levels of depression and anxiety, which again suggests that the escape into an alternative reality does little to resolve the actual distress that someone is facing.

For many maladaptive daydreamers, the fantasies are so rewarding that they take precedence over real life experiences. Consider the words of Pietra: “Nothing else feels as enjoyable.” She says that at one point in her life, she could not go 10 minutes without entering a daydream. “I would go into them no matter what I was doing.” This interfered with her academic studies, her relationships, and even eating regular meals. “I’d postpone my meals by two or three hours while I starved,” she says. “And food was right there to be eaten.”

Such reports have led some psychologists, including Somer, to view maladaptive daydreaming as an addiction, akin to compulsive gambling or alcoholism. “Immersive daydreaming could be like drinking a glass of superb wine,” he says. “But downing a bottle of vodka every day is not good.”

Despite the severe difficulties they are facing, many of the maladaptive daydreamers find it hard to share their experiences with the people around them. “I’ve only told three people and they had similar reactions: they looked as if they wanted to laugh,” Karina Lopez tells me. Michelle agrees that, from the outside, the issues can seem superficially trivial. “It seems like something that you could very much control,” she says. “But trust me: I’ve tried.” For this reason, she says, it has been harder to disclose her maladaptive dreaming than her anxiety and depression, even with the stigma surrounding those mental illnesses.

***

Despite our lack of awareness and understanding, these kinds of experiences are surprisingly common. In a survey of more than 1,000 Jewish Israeli participants, Soffer-Dudek found that about 2.5% of the population met the criteria for maladaptive daydreaming. That’s one in 40 people, which would mean that the condition is more common than anorexia nervosa or obsessive-compulsive disorder, and similar in prevalence to generalised anxiety disorder. While further studies will need to establish the prevalence among larger and more global samples, it seems probable that at least one of your acquaintances will be struggling with an urge to escape into their immersive fantasies.

Intriguingly, maladaptive daydreaming seems to be far more common among people who have been diagnosed with attention-deficit and hyperactivity disorder, with a recent paper reporting a prevalence of about 20%. (Moreover, 77% of people with maladaptive daydreaming have been diagnosed with ADHD.) The constant desire to slip into daydreams, it seems, is contributing to difficulties in concentration and focus – and this group may require different forms of treatment from other people with ADHD.

Given these findings, Somer believes that maladaptive daydreaming should be recognised formally as a disorder by organisations such as the American Psychiatric Association, which publishes the influential Diagnostic and Statistical Manual of Mental Disorders. “We have accumulated a body of evidence to show in the reliability of this construct, and that it cannot be better explained by any other psychiatric condition,” he says, adding that he has already received positive feedback for the proposal.

Musetti agrees that we need greater awareness among health professionals. He says there’s a quickly growing number of people online describing maladaptive daydreaming, but these bloggers often hit a wall when they try to get professional help. “They often won’t find any recognition of their suffering, or a suitable treatment,” he says.

Exactly how maladaptive daydreaming should be treated is an open question – though there are promising signs that people can learn to control their habit. In 2018, Somer published a case study of a 25-year-old undergraduate named Ben who would spend around three hours a day in his fantasies. Ben had originally been diagnosed with ADHD and was given a course of Ritalin, which only increased his tendency to daydream.

Working with Ben to find a potential solution, Somer suggested cognitive behavioural therapy and mindfulness training. Ben would note down the circumstances that seemed to be associated with his maladaptive daydreaming, for example, and prepare careful plans for each day to try to reduce the temptation. And when he found himself falling into his fantasies, he would try to interrupt the daydreams’ plots with unsatisfying endings. By the end of the six months, he had reduced his habit by about 50%.

Based on this success, Somer has since conducted a clinical trial with hundreds of participants. Although the study has not yet been published, he says the results are “very encouraging”.

Both Somer and Musetti agree that in many cases, it may not be possible, or even desirable, for people to eliminate their daydreams altogether; instead, the aim should be to enable them to regulate their habit, and to find alternative ways to process their negative emotions. “They could perhaps confine it to certain times of day,” says Somer.

Kyla, for one, would be reluctant to lose her daydreams completely. While her fantasies had once been maladaptive, they no longer dominate her life. Rather than using the daydreams simply to escape negative feelings, she says she can use conversations with her characters to gain perspective on problems. In one mental health crisis, she believes that this even saved her life. To suppress the daydreams altogether would be impossible, she thinks. “It’s just how your brain works – you can’t just turn it off.”

* To preserve their privacy, Kyla, Michelle and Pietra asked the Observer not to print their surnames

• The Expectation Effect: How Your Mindset Can Transform Your Life by David Robson is published by Canongate (£18.99). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply