What is it about the heart and pop songs? Owner of a lonely heart. Everybody's got a hungry heart. Looking for the heart of Saturday night. The seminal Half Man Half Biscuit song 'I Left My Heart in Papworth General'. No one sings that way about the kidneys or the pancreas.

A few weeks ago at a lunchtime meeting in the elegant minimalist conference room at the Hempel hotel in west London, someone's mobile phone played the opening bars of 'My Heart Will Go On'. Among cardiologists and others in the heart business this may once have passed for a thigh-slapping joke, but on this particular Wednesday anyone within earshot turned away in horror. The many heart experts in the room - consultant professors, executives from the British Heart Foundation, specialist nurses, representatives from the health and medical journals - had gathered to discuss the results of a large survey about heart failure, and the atmosphere was friendly but serious. No one was in the mood for funny ringtones.

The heart people had come together at a time of great excitement in their world. There wasn't very much they couldn't do to repair the most complex of problems. Transplants were routine. Quadruple heart-bypass surgery would have you back on the golf course within a month. Wonder drugs to control the heart rate and thickness of the blood saved countless lives every year. Pacemakers were already stale news compared to the tiny, implantable defibrillators that administered electric shocks and restored a normal heartbeat. Each week all over the country, people dressed as Big Bird ran round parks to raise millions to extend yet further the boundaries of cardiovascular research, in the well-founded belief that what can't be fixed now will be fixed in the future: there was already the prospect of stem-cell breakthroughs enabling muscle patches to be placed on a damaged heart with the ease with which we now place plasters on a knee.

But those gathered at the Hempel hotel had one problem that still caused palpitations. Why, if we know so much about the heart, do we know so little about heart failure? And why are we pursuing the glories of biotechnology while simultaneously witnessing the premature deaths of thousands of people each year from what appears to be nothing more than unwitting ignorance?

Heart failure is what you end up with at the end of your life. The most common cause is coronary artery disease, usually brought on by smoking, obesity, diabetes and high cholesterol levels. If you have a heart attack and survive, there is a large risk you may eventually die from heart failure. If you have angina or high blood pressure, you are also at risk.

Heart failure - which may best be described as the condition that arises when the heart muscle becomes too weak to pump sufficient blood around the body - is the final expression of all forms of heart disease, and its prevalence is alarming. About 14m people in Europe suffer from heart failure, a figure that has been predicted to double by 2020. More than 3.6m cases are reported in Europe each year, and in the UK about 65,000 new cases are expected in 2005. About one-third will die from the disease, accounting for 4 per cent of total UK deaths. Heart failure is more common than most cancers, and more women will die from it than from breast cancer, cervical cancer and ovarian cancer combined.

Heart failure is not just a problem for the elderly, and among the young it carries a stigma. One person at the Hempel conference was diagnosed three years ago in her early forties, and she is still wary of speaking to a journalist about it.

'In November 2002 I started to notice that when I was walking I was becoming breathless,' Helen Ives (not her real name) wrote to me later. 'The spiral staircase at work started to get difficult to climb in the mornings. At first I just thought, "I'm over 40 now and not as fit as I used to be." But after a few weeks this became progressively worse, until eventually I could only climb a couple of stairs before I was completely out of breath.'

She also found sleeping difficult, and breathing became tricky when she lay flat. 'I was very scared about getting help, but three days before Christmas I went to my GP, as my breathing had become so bad that I could even feel a bubbling sensation in my lungs.' A week later a heart consultant diagnosed familial dilated cardiomyopathy, an inherited disease. 'I was only 43 and have two children,' she recalled, 'and I thought my life would never be normal again. I was very frightened to go out on my own at first, as I kept thinking something was going to happen.'

She was told to eliminate alcohol and caffeine from her diet, and to reduce her salt intake. She learned the names of some new drugs: she was prescribed a daily intake of 40mg furosemide (to reduce fluid retention), 2.5mg lisinopril (an ACE inhibitor which helps dilate the arteries), 5mg bisoprolol (beta-blockers to slow the heart rate) and 75mg of aspirin to thin the blood. She takes them all at once after breakfast, and no longer feels dizzy or nauseous at the resultant drop in blood pressure.

'I just want to stay normal as long as possible,' she said. 'Other people initially seemed to struggle with what seems like an extreme diet and lifestyle change, and still sometimes people will say, "Oh just have one glass, or just a piece of this or that."'

Heart failure is what you end up with at the end of your life, but it need not be; it can be easily and inexpensively treated. It is not possible to say how many lives would be saved or significantly prolonged if patients recognised heart failure earlier and if more people we entrusted with our medical care knew how to treat it better, but the figure is likely to be huge. Helen Ives sat quietly at the Hempel hotel as professionals discussed how thousands of people each year fail to spot the three key outward symptoms: extreme tiredness even when at rest, shortness of breath even when relaxed, and swollen ankles. If they do report to their local surgery, they will be lucky if their GP gives them the treatment they need to get well.

The conference was organised by the Study Group on Heart Failure Awareness and Perception in Europe (Shape), an activist body consisting of medical specialists from nine countries that receives financial aid from most of the major drug companies. Shape was established in 2002, and a year later at a meeting of the European Society of Cardiology in Vienna it revealed the results of a survey conducted among almost 8,000 members of the general public, chosen at random, and another 2,000 primary-care physicians. Last month, these results were considered ready for public consumption and repackaged in the starkest of terms.

Among the general public, only 3 per cent could correctly identify the signs and symptoms of heart failure. Almost 65 per cent believed that survival rates were better than those for cancer, whereas in truth they are bleaker, with only 25 per cent of men and 38 per cent of women living for more than five years after diagnosis. Among doctors and other general health carers, the results are equally arresting. About half did not provide the optimum care set out in national and European guidelines. The two leading drug treatments - both cheap and widely available - were insufficiently prescribed; despite clear guidelines, only half of GPs recommended ACE inhibitors as a front-line therapy; and only 34 per cent said they would offer beta-blockers as supplementary treatment, despite the fact that many medical papers have reported that their use reduces mortality by one-third. About 75 per cent relied for their diagnosis on patients' symptoms, rather than using routine but slightly more time-consuming testing methods, such as blood tests or echocardiography.

It was clear that one of the simplest problems to solve was proving surprisingly difficult. Speaking at the Hempel conference, Peter Hollins, the director general of the British Heart Foundation, observed that the great advances in surgical techniques had a downside: there are now far more people living with the symptoms of heart failure than ever before, almost 900,000 in the UK, costing the National Health Service at least £600m. The BHF was continuing an extensive educational campaign to achieve earlier diagnosis and better treatment, and hopefully slow the progression of the disease, and improve survival rates.

Another participant, Dr Roger Boyle, known officially as the national director for heart disease and unofficially as the 'heart tsar', said that 10 per cent of hospital beds are taken up by people with heart failure. As such, the Department of Health regarded it as a key priority, and Boyle spoke of reducing the number of 'recidivists', who returned to hospital on a regular basis. Recently, £60m had been spent on new diagnostic echocardiography apparatus and the more costly magnetic resonance imaging (MRI) machines. There was evidence these efforts were having an effect: hospital admission rates have begun to dip after a huge increase in the past decade, although this decline is much more visible in women than men - perhaps, the heart tsar suggested, because 'men are the weaker sex and give in more easily'. He noted that prescription rates of cardiovascular drugs had almost doubled in the past four years, from £1bn annually to a predicted £1.8bn this year, the large majority consisting of treatments to lower cholesterol.

But when the speeches were over, another expert at the Hempel admitted there was a very long way to go. 'There is still an amazing lack of awareness,' one consultant professor told me. 'If I told you I had a cocktail of two or three drugs and with them the ability to halve the mortality related to breast cancer or lung cancer, you'd think this was important, and you would probably be amazed if only a fraction of eligible people were getting them.'

The history of our struggle to mend the broken heart is almost as old as the heart itself, but the modern chapter starts at the beginning of the 19th century with the French doctor RenÀ Laennec. Laennec was not the first person to note that the heart was not, in fact, heart-shaped, but it was he who popularised the observation that it was roughly the same shape and size as an adult fist. He also made the first accurate calculations of its precise measurements: its weight varied from 230 to 280g in a female, and from 280 to 340g in a male; the average length is 12cm, width 9cm and thickness 6cm; the size and weight continue to grow into old age.

Laennec also developed the first stethoscope as a diagnostic device, although in his day this consisted of a sheet of paper rolled up into a tube. Before him, doctors had either outwardly determined the health of the heart by taking the pulse at the wrists, or by pressing their head to a patient's chest, something Laennec wrote might 'embarrass both parties if the patient were young, female, modest and physically well endowed'. About 100 years later, at the beginning of the last century, the stethoscope was first used to provide a measurement of blood pressure in conjunction with a sphygmomanometer, the air-pressure cuff placed on the arm to cut off the blood flow while an attached column of mercury gauges both the normal and the maximum output pressure of the heart.

These gadgets were the culmination of hundreds of years of trial and error anatomy that gathered pace with Claudius Galenus, physician to Marcus Aurelius and Roman gladiators in the 2nd century, and stepped into the modern age with William Harvey in 1619, when he began demonstrating his theory of circulation at St Bartholomew's Hospital in London. Harvey's analysis of the heart's working relationship with arteries and veins was one of the greatest breakthroughs in medicine, for Harvey saw the whole picture. He cut up human corpses alongside living dogs, pigs, snakes and lobsters, and his conclusions were revolutionary.

He established the heart at the centre of the human framework, whereas previously the liver was dominant. He showed that the heart is a hollow muscular pump with four cavities, each with a valve, the whole contained in a strong fibrous bag and covered with a lubricated membrane that enables its movements to occur virtually without friction. He explained that the contraction of the heart fractionally precedes the pulse, squeezing out the blood rather than sucking it in; the pulse is a result of this action, not an independent muscular expansion of the arteries, as previously thought. He contradicted the belief of Claudius Galenus that the blood in the arteries is different to the blood in the veins. He established the notion of the heart as a natural machine in which 'one wheel gives motion to another, yet all the wheels seem to move simultaneously'.

After Harvey, crucial medical developments followed in a steady stream, but in the past 50 years the pace of change has been extraordinary. The first correct interpretations of blood clotting and coronary-artery disease were made by 1775; a century later the feasibility of blood transfusions was discussed, and a link made between diet and atherosclerosis (the furring of the artery walls with fat deposits); in 1887 the British physiologist Augustus Waller invented the electrocardiogram to measure the heartbeat; and the first artificial heart was developed in 1929 at Dalhousie University in Nova Scotia, consisting of two bellows within a round brass container. In 1952, the first implantable pacemaker was developed and was commonly implanted a decade later; in 1957, non-invasive ultrasound was used to view the heart on a monitor; in 1967 the first coronary bypass surgery was performed using a patient's own veins, and the South African surgeon Christian Barnard performed the first heart transplant in Cape Town (the first transplant in Britain occurred the following year); in 1968, the first balloon was placed in a living patient to aid pumping; in 1969 the first artificial heart was implanted to act as a temporary bridge while a donor heart could be found; in 1984, a baby's heart was replaced with one from a baboon, which functioned for three weeks.

In recent years, our ability to treat all forms of heart malfunction has led to the inevitable moral complications about the treatment's desirability. The arguments surrounding the possibility of keeping a patient alive indefinitely continue to occupy high courts throughout the world, but new debates have extended to stem-cell research and cloning, as well as other issues quite unimaginable in RenÀ Laennec's day. At the end of last year, a drug called BiDil was found to have a beneficial effect on heart failure in a certain group of patients with insufficient nitric oxide in their blood vessels. The dilemma was, the patients were all African-Americans. White patients did not prosper nearly so well on the drug, thus raising the possibility of a heart medicine being licensed and marketed exclusively for those with a certain skin colour. Although BiDil is not a new compound, its specific efficacy has only recently been noted. The political ramifications of launching what some will regard as a dangerous step towards 'pharmaco-genetics' are yet to be felt.

In other labs, other excitements. Last December, researchers at MIT announced the possibility of producing beating heart tissue that may be patched over damaged heart muscle cells. Experiments in rats successfully replicated the natural function of heart muscle cells that are damaged during a heart attack and from other diseases, something previously unattainable.

There have also been great advances in artificial hearts for those unable to get human transplants, with a growing belief that the implantation of these slender titanium cylinders placed alongside a diseased heart may soon eliminate the need for transplants at all (the waiting list for transplants in the UK is about 300; the annual worldwide demand for artificial hearts has been estimated as anything from 20,000 to 200,000.) The effectiveness of implantable cardioverter defibrillators - a pulse generator and battery pack about the size of a packet of gum fitted near the armpit to monitor the heartbeat and deliver corrective jolts - has made them a diminutive personal alternative to those shiny shock pads we see on every episode of Casualty after someone shouts 'Stand clear!'

The everyday treatment of heart disease has seen similarly spectacular developments in the past five years. The British Heart Foundation estimates there are 147,000 heart attacks per year in men of all ages, and about 121,000 in women. About 66,000 men have heart attacks under the age of 65, as do 26,000 women. The incidence rate for both sexes is between 2 and 2.5 times the mortality rate. Those who do not die from their first attack have a substantially extended expectation of life compared with a potential prognosis a decade before, predominantly due to the improved therapies they take after breakfast.

Few heart patients will not be aware of the impact of statins, the cholesterol-lowering drugs prescribed to 1.8m people in the UK and estimated to save 6,000 premature deaths a year. This summer, simvastatin will be available for the first time over the counter after a brief consultation with a pharmacist, although there are fears that possible side-effects - including an adverse reaction if a patient drinks a lot of grapefruit juice - may be ignored. One can, of course, already purchase the blood-thinning properties of aspirin, a drug with so many uses for its salicylate compound that it has recently been called vitamin S.

In addition, a new diagnostic test is causing a mixture of elation and confusion in the United States. Two studies published this January in the New England Journal of Medicine suggest that blood levels of C-reactive protein (CRP) are as accurate a predictor of heart disease as cholesterol. Instead of measuring fat in clogged arteries, CRP gauges the levels of arterial inflammation. The good news is, there are already anti-inflammatory drugs on the market known as COX-2 inhibitors, but the bad news is that they come with the most severe warnings. Last September, Merck withdrew its COX-2 compound Vioxx after many reports of related heart attacks in the United States; a few weeks ago The Lancet reported the figure stood at between 88,000 and 140,000 cases.

The biggest recent benefit to both patients and GPs with regard to heart failure is a new and easy test for patients presenting with swollen ankles and breathlessness: the natriuretic peptides screening. These peptides are small protective chemicals released by the heart under stress, and when the heart is failing it produces these chemicals in excess as a protective response. Levels can be measured from a pinprick of blood placed on treated paper, with results detected within 10 minutes. If the particular b-type natriuretic peptide (BNP) blood level is not elevated, then you haven't got heart failure, and you may just be overweight and unfit, or have asthma.

But if the BNP level is high, you should have an echocardiogram. BNP measurements also present a potential for a treatment that may mimic their action on the heart, a process that relaxes the blood vessels and stimulates the kidney to secrete salt and water, thus replicating the body's attempt to overcome all the things that occur when heart failure develops. In time, such a treatment may be fine-tuned to each patient's precise requirements. So we may again ask how it is that we have come so far without being able to control the rapid rise of the most basic of complaints. There are few people better equipped to answer this than John McMurray, professor of medical cardiology at the University of Glasgow, the author of the World Health Organisation's guidelines on heart failure and many books on the illness. When we met, his frustrations were balanced by some optimism; the problems were serious but not out of control.

I asked him who was accountable for this situation. He said, 'I have to say, cardiologists should take some responsibility. Most aren't particularly interested in heart failure. They're more interested in coronary artery disease problems that precede it. Many of my colleagues would be interested in doing balloon angioplasty and procedures like that, which are very important in their own right.

But we have so few cardiologists in the UK, and if people express a preference for the management of coronary artery disease, then heart failure has been neglected. People aren't talking about it, aren't interested in it, haven't provided services to cater for it. It all trickles down from there, because if enough people aren't interested in it then governments aren't interested, because they have limited resources to spend and are not under pressure. The people who make the most noise get the most money.'

McMurray says he was surprised by the lack of knowledge shown in the Shape survey, not least in the response to one question in particular. Having failed to identify the symptoms of heart failure, most people also said they would not regard them as important, instead treating them as a natural sign of getting older. Inevitably this would mean delaying appointments for consultation, and a severe worsening of the condition. About 40 per cent of those affected die within a year of first hospitalisation, which hardly describes the perceived condition of a patient undergoing a sedate decline into old age. McMurray was also concerned at the under-use of effective treatment. Both ACE inhibitors and beta-blockers are an essential component of what he calls 'beautiful' guidelines from the National Institute for Clinical Excellence (Nice), but the recommendations have not had much impact in many GPs' surgeries. This may be because, despite the overwhelming evidence of their current effectiveness, the drugs have an unfortunate history.

When he was at university, McMurray was taught that beta-blockers shouldn't be given to patients with heart failure. The problem was the dosage: when the drugs were first administered, they were given at the same potency as that for high blood pressure, and could kill the patient on the spot. As with chemotherapy, beta-blockers and ACE inhibitors must be delivered in a very precise way, and the challenge is to overturn a generation of fear and inexactitude. The rewards can be high. 'I don't think we've told the true story yet of what a difference they can make,' McMurray told me. 'Not very long ago I looked at a man's heart a year after he'd been on beta-blockers, and I said to the technician doing the echocardiogram that I really couldn't believe it was the same patient, and was she sure she hadn't mixed up the pictures with someone else? The heart had gone from extremely abnormal to normal, and that's not an uncommon experience. That's why transplant waiting lists are declining, because beta-blockers make such a difference.'

We may reasonably ask why are we thinking about all this now. It is because we can do so much more about it than we could even five years ago. The drugs are inexpensive enough to enable anyone to live a decent and extended life, and the implants and surgeries are highly feasible, but it all begins with accurate and early diagnosis.

Unlike HIV and other more arresting afflictions of our age, heart failure has few celebrity advocates, and its chronic, progressive course argues against dramatic health campaigns. But it should be as vivid in our minds as the drama of a heart attack (in fact, those who survive a heart attack will probably have a better prospect of long-term survival than those who present with late-stage heart disease). Although there are finally some positive signs that heart failure is being treated with the urgency it demands, the incidence is set to worsen dramatically as the average age of our population increases. So what can we do to reverse the trend? Oh, the usual: quit smoking, eat more healthily, don't ignore the warning signs. Heart failure is what we end up with at the end of our lives, but at no other point in our lifetime have we had such a clear opportunity to redirect the course of our own medical history. Which is another way of saying that if we don't do something about it now, we're going to be in big trouble.