My first serious sleep problem was triggered by looming redundancy. Faced with the prospect of losing my job, worries over money just before Christmas and anxiety about the future, I lay awake for hours, my thoughts racing, or wandered around a freezing house, munching bowls of cereal.

I'd finally drop off about five, only to have to get up at seven. This led, in turn, to days spent in a haze of nervous exhaustion and fatigue, just at a time when I needed all my energy to stay positive about the future rather than sinking into a pit of misery.

Sleeping pills made life more tolerable at first, but turned out to add to my troubles in the long term. Advised by the GP to take the pills "as needed", it was easy to convince myself I couldn't get to sleep without them. Soon I was taking them every night and starting to panic that I'd become dependent. I finally stopped taking them on New Year's Eve – and, distressingly, found myself awake the whole night.

Just over a year later I've recovered and am, for the most part, sleeping like a baby. But everywhere I look there's someone with a sleep problem. My husband lies awake thinking about his business, looking corpse-like each morning; a college student I know often doesn't drop off till three or four; a psychotherapist friend tells me that four years ago, afflicted by money worries, she started to wake regularly at 4am and has done so ever since; a friend of a friend says she often goes whole nights without sleep.

"I'm retired, so life doesn't fall apart," she says. "But the next day I stagger around on automatic pilot. I'm not safe to drive and I never feel like going out."

Are we becoming a nation of insomniacs? Are we in the midst of an epidemic of sleep disorders, caused by stress at work (or by the lack of work), soaring anxiety levels and the pressures of a 24/7 lifestyle? It would seem so, judging by a survey of more than 6,000 British adults, in a report published last week by the charity the Mental Health Foundation. It found that nearly two-thirds have problems getting a good night's sleep while about one in three may suffer from chronic insomnia, affecting mental and physical health, mood, energy, concentration levels, relationships and day to day functioning. Insomnia, it says, is a "massive public health problem", the most commonly reported mental health complaint in the UK.

"Obviously a survey like this attracts people with sleep problems," says Professor Colin Espie, director of Glasgow University's sleep centre and co-founder of Sleepio, a new organisation dedicated to raising awareness about the importance of good sleep, which collected the survey data. People with chronic insomnia – usually defined as having sleep problems three or more nights a week for at least six months, causing distress and affecting daytime functioning – probably form 8-10% of the population and possibly 15% of those over 65, he says. Still, there are clearly many more who are troubled by episodes of sleeplessness, which can develop into chronic problems, he warns.

So why is sleep so crucial? In humans, its main purpose is to allow the brain – in particular, the cortex, the region that plays a key role in memory, perception and thought – to recover and regenerate, enabling us to learn and function during the day. In most people, sleep is regulated by the circadian rhythm, the internal biological clock that responds to light and darkness, so that (hopefully) we sleep at night and are awake during the day. This explains why for many people shift work is a problem and is thought to be linked to a greater risk of cancer and heart disease. How much sleep individuals need varies, with some naturally needing less than others, although research at California University has found that people who live longest report sleeping for six to seven hours each night.

What happens when we can't get to sleep? Espie describes insomnia as a "disorder of arousal". "It's a little like an overrun when we switch off the car engine but it keeps on ticking over before cutting out. People with insomnia have higher levels of arousal at all levels, both of the mind and body, and find it difficult to shut down – hence the racing thoughts."

Initially, sleep problems are usually triggered by an external, stressful event such as illness, bereavement or (as in my case) redundancy. But while most people can repay the "sleep debt" caused by sleepless nights and eventually get back to normal, in some people insomnia becomes entrenched, continuing even when the initial stress has disappeared or been resolved.

"Some people just seem wired up to be more vulnerable to insomnia," says Professor Kevin Morgan, from Loughborough University's sleep research unit. While individual risk of insomnia varies, he adds, there is good evidence that social changes and, in particular, economic recession is linked not only to higher levels of anxiety but an increased incidence of insomnia.

"One advantage of a daytime job is that it helps our bodies stay synchronised to the 24-hour clock. Without the need to get up for a job or go to bed, our biological clock can become weaker."

In the short term, lack of sleep makes us weary, apathetic, forgetful and irritable; in the longer term it is linked to impaired performance, job problems, mood disorders and mental health problems such as depression (although depression can also lead to poor sleep); while research from the US has found it a risk factor for obesity, high blood pressure and depression. Chronic insomnia may affect the immune system.

While sleep experts agree there is an urgent need to address the problem, sleeping tablets – for which, says Morgan, there are about 10m prescriptions annually– are definitely not the answer. They may be useful for short-term sleep problems but, used for more than a few weeks, they can cause withdrawal symptoms when stopped – the principal ones being increased anxiety and more severe insomnia. Instead, say sleep specialists, people with sleep problems should be offered cognitive behavioural therapy (CBT), a form of talking treatment that focuses on changing the thinking and behaviour that stops people from sleeping – helping insomniacs, for example, to give up trying to fall asleep; or enabling them to view their beds as places of comfort and relaxation.

There is now good evidence that CBT (which is also used for problems such as phobias, depression and anxiety disorders) can work for insomnia, with 70% of sufferers showing some improvement; but availability is a "postcode lottery", says Dr Andrew McCulloch, chief executive of the Mental Health Foundation. The charity is campaigning for improved access to CBT for insomnia, as well as for Nice (the National Council for Health and Clinical Excellence) to develop guidance for doctors on using non-drug therapies for insomnia, more training for GPs on the importance of sleep, and research into the effectiveness of CBT-based self-help courses and books.

While most stress has been placed on a psychological approach, recent laboratory findings could help develop effective treatments for insomnia in the future. Scientists at Cambridge University have found that the biological clock that regulates the activity of all cells in a 24-hour cycle works independently of DNA, or genetic programming.

"Every single cell in the body has its own biological clock which maintains the circadian rhythm," says Dr Akhilesh Reddy, a clinical neurologist at Cambridge University. "We took some human red blood cells which contain no DNA and found that, even in the absence of genetic information, there are enzymes in the cell, called peroxiredoxins, which causes chemical oscillation on a 24-hour cycle." The same held true for single-celled green algae, when scientists blocked off gene activity. "Despite being separated from humanity by one billion years of evolution, the algae showed the same common 24-hour cycle."

The discovery, published in the scientific journal Nature last week, could enable scientists to develop drugs which directly target this same protein in brain cells – a far easier task than developing drugs which target DNA. Reddy says it may now be possible to develop drugs that make the "brain clock" stronger in people having sleep problems.

"These types of drugs wouldn't knock people out like sleeping tablets," he says. "They would be curative, rather than putting a plaster over the problem."

Above all, say sleep experts in all fields, a public health strategy to help prevent sleep problems is urgently needed, on a par with campaigns for healthy eating, exercise, alcohol and smoking.

"When I was younger we didn't know about the importance to health of a good diet or exercise, but we now take that information for granted," says Espie. "Sleep is also crucial to our wellbeing – and it should be on the public health radar."