Last year, 15.3 million NHS prescriptions were made for sleep medication. Patients in England alone received 5.4m prescriptions for zopiclone and 2.8m for temazepam, the two most popular sleeping pills. According to the Economic and Social Research Council, one in 10 of us now regularly takes some form of sleeping tablet.

But last week, a leading sleep specialist – Kevin Morgan, professor of gerontology at the University of Loughborough – argued that the NHS should be looking beyond sleeping pills and training its staff to provide psychological therapies such as cognitive behavioural therapy (CBT) to help those with insomnia. Morgan argues that CBT offers a better long-term solution, and that sleeping pills hold further dangers: addiction or dependency, medication-related accidents, and spiraling costs for the health service.

So how easily can we be weaned off them? Our desire to medicate ourselves in this way is part of our wider preoccupation with sleep. Sleep is perhaps only rivalled by sex in terms of our concern about how much we get, how much we want and how we go about doing it. Earlier this year, when Nuffield Health reported that UK adults "miss out" on 378 million hours of sleep a week, their respondents averaging 7.1 hours a night, they stressed that this figure was "much less than the recommended eight hours". The idea that we might not be attaining the desirable number of sleep hours fuels further sleep anxiety.

In 2006, Forbes magazine published an article called The Sleep Racket, detailing the billions we spend on mattresses, pillows, herbal sleep-aids, soothing room-spritzers, bath-soaks and other products each year. Increasingly businesses are capitalising on our worries about what might happen if we don't get enough sleep. We will, it seems, get fatter, quadruple our risk of stroke and increase our chance of a car accident. Recently, the American Academy of Sleep Medicine reported another impact of sleeplessness – it causes a spike in white blood cells to rally the immune system, in much the same way as the body reacts to high stress.

The effect on the body is noticeable even after a single restless night: brain-scan research at the University of California showed that the amygdala, the area of the brain responsible for emotional response, overreacts when shown disturbing or upsetting images. This emotional stress soon translates to physical stress, which affects the immune system and leads to inflammation; the American Heart Association found that sleeping under six hours a night causes inflammatory substances in the blood to increase by 25%, as well as raising blood pressure and heart rate and affecting glucose levels. If this is maintained for some time, it can result in hardened arteries and increased risk of infection.

In light of this, are sleeping tablets really so terrible? Might they work to simply realign our sleep patterns, to break the vicious cycle of the insomniac? "They do have a place in treatment," says Dr Andrew Hall, consultant in sleep disorder medicine and a member of the British Sleep Society, "especially for an acute problem, such as a grief reaction, when it may be a humane act to prescribe them."

Professor David Nutt, director of the neuropsychopharmacology unit at Imperial College, goes further: "The hope is you knock people back into the routine of sleep," he says. "Some people, whether for genetic reasons or stress, have persistent trouble sleeping, and some of those people need sleeping pills in the long term."

Yet for all our increasing reliance on medicated sleep, there lingers a residual social suspicion of sleeping tablets – fired, perhaps, by the memory of barbiturates (usage of which only began to dwindle in the 70s and 80s) and of the zombified "mother's little helper" image of Valium.

It was in the early 1950s that a Polish immigrant named Leo Sternbach was employed as a researcher at the headquarters of drug company Hoffman-LaRoche in New Jersey. The postwar US had displayed a considerable appetite for anti-anxiety medication, and Sternbach had been charged with the task of creating a copycat version of Miltown, the most-popular anti-anxiety drug of the day. Sternbach instead decided to devote his time to pursuing his own investigations. It was four years before he and his colleagues felt they had found something of note, and tested the new drug on laboratory mice, observing the creatures' behaviour when placed at the bottom of a sharply inclined screen. Previous experiments had shown that un-drugged mice tended to clamber up the screen with ease; drugged mice were likely to slide back down the screen to join their equally sluggish comrades at the base. With the new drug, researchers noticed that although the mice also slid back down the screen, at the base they remained quite normal and responsive. Sternbach's discovery of benzodiazepine would transform the realm of sleeping tablets from the crude mechanisms of its predecessors.

The first commercially available benzodiazepine drug was named Librium, and was approved in 1960, though it was soon eclipsed, three years later, by his improved version: Valium, which became the biggest-selling drug in America between 1969 and 1980, with 2.3bn sold in 1978 alone. "It's quite a good drug," Sternbach told the New Yorker, at a 40-year anniversary party thrown in Valium's honour. "It has pleasant side effects. It's quite a good sleeping tablet, too. That's why it's abused. My wife doesn't let me take it," he added tellingly.

The science of sleep medication has evolved since the work of Sternbach. Temazepam – along with flurazepam, loprazolam, lormetazepam and nitrazepam, is part of the modern class of benzodiazepines. Zopiclone is one of a group of medications known as Z drugs – zaleplon, eszopiclone, and zolpidem (better known as Ambien), which are not benzodiazepines, but function in a similar fashion, reducing the length of time it takes the patient to fall asleep and extending that period of sleep. Where they ace benzodiazepines is that they do not affect "sleep architecture" – that is the structure and pattern of sleep, and the duration of REM and non-REM sleep. Benzodiazepines reduce deep slow-wave sleep, so that we do not wake feeling particularly rested. Neither group of drugs are designed for long-term use.

Ian Singleton is 65, and a senior project worker at Bristol Tranquilliser Project. The project helps an estimated 250 people a year, about half of whom are dependent on Z drugs. Singleton himself became dependent on benzodiazepines in the 1980s, having visited his GP about recurrent stomach upsets and work-related stress. "My doctor diagnosed stress," he recalls. "But he didn't ask why I was stressed, he just said he would give me drugs to help me get back to work."

At first Singleton thought the drugs might be helping. "I felt not too bad for about a month," he remembers. "Then I could feel the effect wearing off. I got pains in my body, terribly tight stomach muscles, and I was panicky. I knew after a year that I was hooked." The effect was devastating. "It completely wrecked everything," he says. "I couldn't sleep, I had anxiety, I had no interest in life. I felt like a bomb ready to go off."

Singleton went cold-turkey, experienced an epileptic fit and was hospitalised for several months. "I would never advise anyone to take sleeping pills," he says. "Unless they have been through a traumatic event and the doctor has prescribed them for three or four days." There is always, Singleton argues, a reason why people don't feel able to sleep. "Drugs should only be prescribed when everything else has failed."

It costs the NHS around £50m a year to keep us in sleeping pills. The demand has risen since the dawn of the recession. In May, the NHS Business Services Authority, which pays pharmacists for prescriptions, revealed a 10% increase in prescriptions in the east of England over the past four years, at a cost of an extra £7m annually.

This is a great deal of money for a medication that may not be as effective as we hope. Although most patients believe they accrue an extra 50 minutes of sleep by taking them, the American Journal of Public Health reported that Z drugs increase our sleep by a mere 12 minutes. There are other concerns, too. In February, BMJ Open published perhaps the most alarming argument against the use of sleeping tablets. An American study compared 10,500 patients taking prescribed sleeping pills with 23,000 patients not on sleep medication. Although the absolute risk of death remained quite low, it was four times higher in those taking sleeping tablets.

"What was new in our paper was information about association with mortality and cancer of specific drugs, including the agonists zolpidem, zopiclone, the older benzodiazepines, barbiturates, and dipheniramine," says Dr Daniel Kripke of the Scripps clinic in California, and the report's lead author. "They all appeared associated with seriously increased mortality. All appeared associated with cancer also. One of the disturbing things about our data was that even taking fewer than 18 doses per year appeared associated with excess mortality."

He is keen to stress that much of this was seen in patients who had other co-morbidities such as obesity, suggesting the possibility that such conditions might interact with the dangerous effects of hypnotic sleeping tablets. Kripke is adamant about the implications of his research, though: "I conclude that none of the drugs we were able to study are safe enough to use," he says, "and all should be avoided, even for occasional use."

Professor Nutt is sceptical of this research. "There's been very little evidence over the last 60 years that these sleeping pills do any harm," he insists wearily. "Do I think sleeping tablets are a good thing? I sure do. People who criticise them don't treat people with insomnia – people who haven't slept for weeks and it has destroyed their life. It can drive people to depression, to suicide, to take more dangerous drugs, like alcohol. Insomnia is a serious illness."

He concedes there may be drawbacks to long-term use: "People don't get addicted to sleeping pills, but they do get dependent on them, and so they may experience poor sleep when they stop taking them. Part of that is chemical, and part of that is because they haven't addressed the underlying problem that has been affecting their sleep; they've just been wallpapering over it."

Last year, the Mental Health Foundation reported that 37% of people claim to suffer insomnia, and just 39% describe themselves as "good sleepers". So how to remedy our sense of disjointed sleep? Perhaps the first step is to consider how we define good sleep. Nuffield Health's mention of the "recommended eight hours" is something of a nonsense – some people simply don't need as much sleep, while others may need more. The important point is to learn one's own rhythms and requirements. It may even be soothing to remember that the idea of a solid eight-hour block of sleep is a relatively recent invention – previous generations were at ease with the idea of a waking hour in the middle of the night, when people would read, chat, have sex. Around the 19th century, indeed around the time when sleep became increasingly medicalised, we lost touch with this idea of a two-part sleep.

"The best approach to sleep problems is good sleep hygiene," says Hall, "which means how you organise your sleep: no over-stimulating activities too late at night, no computers or television, exercising earlier in the day, a little alcohol sometimes but not too much, and nothing with pilot lights, or bleeping, or things telling you that you have new messages in the bedroom."

The shift in thinking towards treatments such as CBT aims to empower patients, enabling them to take control of their own sleep problems, and save the long-term costs of sleeping tablets. The Department of Health says it has assigned £400m over the next four years to improving access to this kind of talking therapy.

CBT works on the basis of understanding and manipulating the two processes that affect an individual's sleep – first, the learned behaviour of spending too long awake in bed trying to get to sleep, and second, the thinking that accompanies that time, which can become panicked and obsessive. Morgan argues that a mere five hours of CBT can deliver benefits in more than 70% of cases. "It gives a better quality of sleep up to a year later. Thirty years of sleeping tablets have created passivity, the idea that it is out of your control."

That has to be a positive aim. After all, this is not only about health, about increased risk of mortality, stroke and obesity; it is also a matter of whether we want to spend our lives like Sternbach's laboratory mice, forever sliding sluggishly back down a steeply inclined screen.

Even so, sleeping tablets are perhaps not quite the demon we have long painted them to be. They may offer a vital tool in helping the chronic insomniac function, or a simple solution to stop a period of disrupted sleep escalating into something more desperate. As the banner that hung above the heads of those at the Valium 40th birthday party read: "Thanks for the happiness and relaxation you have given us over the years."