Poor sleep – as anyone who suffers from it knows – can make life a misery. And it is taking its toll on the nation's health. According to the recently published Great British Sleep Survey, more than 51% of us now struggle to get a good night's sleep, with women three times more likely to be affected than men.

Evidence collected from 20,000-plus adults between March 2010 and this June shows that 93% of insomniacs report low energy levels and 83% complain of mood swings. Some 77% find it hard to concentrate, 64% say they are less productive at work, and 55% report relationship difficulties.

Even worse, persistent poor sleep can increase the risk of developing conditions including diabetes, depression, high blood pressure and strokes. Research at the University of British Columbia suggests every hour of sleep lost at night may cost us one IQ point the following day. And it is often a long-term issue: a quarter of people with insomnia have suffered from it for more than 10 years.

In Britain, overwhelmingly, we treat poor sleep with medication: the NHS spent a staggering £50m on sleeping pills last year, with 15.3m prescriptions dispensed across England, Scotland and Wales (up 17% in three years). But many pills have undesirable side-effects and the survey's findings suggest they do not solve long-term sleep problems: 42% of people who have taken them on and off have continued to have poor sleep for more than a decade.

So how can we do something about poor sleep without pills? Most people focus first on what Colin Espie, professor of clinical psychology and director of the University of Glasgow Sleep Centre, calls "sleep hygiene": our pre-bed routine, and the physical environment in which we try to sleep. Espie believes these factors account for a mere 10% of sleep problems: "most people with insomnia have better sleep hygiene than easy sleepers". But most sleep experts concur that the following do make a difference.

Light. A dark room is important to a good sleep. Also try to avoid "blue light" less than two hours before bed: research by the Lighting Research Centre at the Rensselaer Polytechnic Institute in New York State suggests light from laptop, tablet and smartphone screens tricks us into thinking it is daytime and keeps us alert, although this has been disputed. Bedrooms should be a comfortable temperature (around 18C), quiet and well-ventilated, with comfortable beds and pillows

Food, drink, exercise. Anything that stimulates the system – such as caffeine (although some experimental studies show a cup of coffee may lengthen the time taken to drop off by just three minutes), alcohol, chocolate, tobacco, a heavy meal or strenuous exercise – will make it harder to get to sleep. Indigestible foods are obviously best avoided; carbohydrates can promote serotonin, which aids sleep. Aim for a regular, balanced diet and no late-night excess. Twenty minutes a day of exercise will make a big difference to your sleep, but avoid it just before bed.

Sleep debt. A weekend lie-in or afternoon snooze can do more harm than good. According to research at the University of Texas Southwestern Medical Centre, "sleep debt" is best "repaid" by getting up and going to bed at your normal times rather than disrupting your body clock. Save sleep for bedtime: naps are recommended only if you are too exhausted to function.

Age. Not strictly sleep hygiene, and not much you can do about it either, but it may help to know that it can get harder to sleep as you get older. Research by the University of Washington Medicine Sleep Centre and others has shown our "sleep architecture" changes as we age: older people spend less time in deeper, non-REM sleep, and their whole body clock or circadian rhythm moves forward.

"Sleep hygiene" alone, however, will not determine whether or not we sleep well. Ninety per cent of the battle is in the mind, which is why talking therapies and cognitive behavioural therapy (CBT) are coming to be seen as perhaps the most useful solution. Espie helped launch Sleepio, a pioneering online CBT programme that has won praise from the medical press, including the Lancet and the peer-reviewed journal Sleep. In a full clinical trial, it helped 75% of people with long-term poor sleep.

So, get your head right, and you will usually sleep. But how? These are Espie's top tips:

Pillow talk

1. Recognise that "sleep is not a lifestyle choice, it is a biological inevitability. It is very, very powerful, and we need to get it working for us: a person not sleeping well has first and foremost to allow sleep to do its work. Sleep is a process of letting go."

2. Sleep's achilles heel, however, "is our world. Most sleep problems are psychological obstacles that we put in sleep's way. Like all things we should do automatically, when we deliberately try to do them we screw up. Adopt a relaxed, confident approach to sleep, not a neurotic, panic-stricken one. Recognise there's a right and a wrong kind of effort. Your role model is the 'careless sleeper'."

3. Go to sleep only when you feel sleepy and, if anything, shorten the time you try to sleep: "A lot of people put good sleep beyond the achievable simply because they are so worried about not sleeping. Their sleep becomes frayed, even more broken. If I try to read a book in bed, it's never very successful because I go to bed when I'm ready to sleep. It's counter-intuitive, but a shorter sleep often means a better quality of sleep."

4. Put the day to bed long before putting yourself to bed. "A racing mind – what happened today, what's on tomorrow, what will the future hold? – is a huge obstacle to sleep. So take time before bed: go through the day, think about tomorrow, put things in their boxes, make a list. Set your mind at rest."

5. Learn to value relaxation, and if necessary learn specific relaxation techniques. Above all, "Don't try too hard." But, of course, for some that is easier said than done.

For more information about Sleepio, visit sleepio.com

• Do you suffer from insomnia? Share your own experiences and tips on how to combat it below