The humiliation is the stuff of every woman's nightmares - the plastic gloves, the cold metal ratchet smothered in KY jelly, the ludicrous "dignity sheet" which hides your indignity from no one but yourself, and of course that hideous scraping. Worst of all is the cheery nurse, chit-chatting about the weather as if poking a spreadeagled stranger's cervix were as mundane as stepping on someone's toe at the supermarket checkout.

Yes, the smear test is truly awful, and waiting for the results can be nerve-wracking too. Even when they come back negative there's always a niggling fear - thanks to those much-publicised lab blunders - that your smear was checked by a particularly dozy work experience student with severe myopia.

In spite of all this, 4m British women take the smear test each year - and the results are undoubtedly a stunning success. Although there are 5,000 new cases of cervical cancer a year, and 2,000 deaths in this country - many among young women - the death rate has fallen by 42% since 1994. That means the screening programme is saving around 1,000 lives each year.

Last week, however, the department of health announced three pilot studies to screen women for the sexually-transmitted human papilloma virus (HPV), which has been proved to cause - or at least be a factor in - up to 99% of all cervical cancers. If you don't have the virus, you're highly unlikely to get the cancer. Vaccines to prevent HPV infection, and anti-viral drugs to fight cervical cancer, are both on the drawing board, but the screening, which will start at three sites around the country this summer, will have the most immediate effect.

If it proves cost-effective, the test, which screens for the virus in material taken during normal smear tests, could go nationwide within two years. This would let doctors identify patients at greatest risk and monitor them carefully.

In the pilot study, women with mild or borderline smears will be tested for HPV. If the results are positive, they will be sent for a colposcopy, where the cervix is examined under high magnification. Any areas showing potentially malignant cellular change can then be removed. If they test negative, they will have a repeat smear and HPV test six months later. If the abnormality has not worsened and the HPV results are still negative, they will go back to the normal screening cycle.

Crucially, it is thought HPV screening could also improve the cancer-spotting accuracy of cytology labs by up to 50%, because cytologists would be prompted to look more closely at the smears of patients who are HPV-positive, and therefore at higher risk.

But if everyone agrees the test has massive potential, why isn't it being made universally available, and why are pilots only starting in 2000, when a good HPV test has been around since 1991? As ever with the NHS, the answer relates to cost. The DNA screening and additional colposcopies will have to prove their worth in terms of lives and money saved on surgery and radiotherapy. Smear testing already costs the government £132m per year, and HPV screening, as well as being more expensive, will probably raise costs still more, because improved detection will lead to a higher initial incidence of cancer.

Specialists point out that the test is still not widely available anywhere else in Europe. "There are other issues to look at, too, for example how women might cope with the anxiety of discovering they have the virus, and then going through the extra tests, when in all probability they won't ever go on to get cancer," says a department of health spokesperson.

But Dr George Walker, medical director of WellCare Lifestyle Clinics, which recently started offering a private HPV test, takes a different view. "The test offers peace of mind," he says. "A lot of women have changes in their smear result, and are told to come back for a repeat smear six months or a year later. They may spend all that time worrying, fearing the worst, and feeling helpless.

"If they have the HPV test, a large majority will find out they are not at risk, and can stop worrying. However, women with the high-risk strain could be sitting on a timebomb, in which case their doctor will ensure they are monitored much more carefully." So far only WellCare, Marie Stopes clinics, and BMI hospitals are offering the test pri vately - all at more than £100 a throw.

So, should women be panicking, shelling out for the private test, or moving to one of the health authority areas (still not chosen) where the pilots will take place? A British Medical Journal report last December said it "may be prudent" to take the test, but Dr Jo Reynolds, of the Cancer Research Campaign, thinks not. "We need the results of the pilot schemes to decide on its efficacy," she says. "Right now, the most important thing is to get women to have the regular smear test. Unfortunately, bad publicity has caused a lot of fear. Half of all women currently dying of cervical cancer have never had a smear." She says regular smear tests can usually identify cervical cancer at an early stage, with 69% of women still alive five years after treatment.

Dr Paul Sasieni, a senior epidemiologist at the Imperial Cancer Research Fund, welcomes the pilot studies, but says: "The government should have been much more pro-active in funding research into HPV. It was clear for a long time that screening would have enormous potential, but the funding was almost non-existent."

Meanwhile, the search for a cure for cervical cancer continues. Some scientists predict the virtual elimination of the disease within a generation; for girls born today, the smear test could come to seem as bizarre as Elizabethan dentistry.

Dr Sasieni is enthusiastic about anti-HPV vaccines, which are being tested in several international trials. Although larger trials are still needed, it is likely that schoolchildren (boys included) or even babies (to eliminate ethical debates about the age at which children start having sex), will be vaccinated in the not-too-distant future to help stop the spread of HPV. Meanwhile, scientists at York University have just finished an eight-year study to map the structure of the virus. This will allow the design of a drug to stop it from reproducing, and will be taken by women with the early stages of cervical cancer. Lead scientist Dr Norman Maitland says: "A drug may be available within 10 years, in which case we can say that cervical cancer will be almost wiped out."

This may be good news for very young girls. But what about women aged 30 and above, who run the highest risk of developing cervical cancer? Dr Maitland says: "Screening is already very good, and cervical cancer is really quite rare. Incidence is falling all the time."

He is even up-beat about lab errors. "Mistakes happen, but Britain is no worse than anywhere else. Because nearly all smears here are done by the NHS rather than privately, we just get to hear about the mistakes more often."

• There are around 80 strains of the HPV virus. About 30 of them are associated with genital tract infections, and 10 with cervical cancer. Very few other cancers have proven to be linked to a virus.

• Women who have had many sexual partners, or start having sex early, are at greatest risk of developing cervical cancer through HPV infection. Around 15% of young women are infected with the virus at any one time, but most fight it off within a couple of months, never even knowing they were infected. Only about 2% of these women will ever go on to develop cervical cancer.

• The virus seems to gain a foothold in those with a lowered immune response, and works by inserting its own genes into the DNA of cells in the cervix, causing them to divide uncontrollably. Other factors include smoking, other cervical infections and poor hygiene. Poorer women also run a higher risk, because they are less likely to have smear tests. Unlike breast cancer, there is little or no genetic predisposition to cervical cancer.

• Britain's cervical cancer rate is similar to most other developed countries even though American women are advised to have smear tests on a yearly basis, while some British health authorities only suggest a test once every five years. This is because cervical cancer generally spreads slowly. However, it spreads faster among younger women, in rare cases causing death within one or two years.

• Rates of cervical cancer are higher in southern and eastern Europe, either because of greater promiscuity or earlier sexual development and childbirth. Israel has abnormally low rates because of attitudes to pre-marital sex.

• Some studies have suggested there was a peak in the number of cervical cancer cases in the 80s among women who had been actively involved in the "free love" era of the 60s.

• The risk of cervical cancer-related HPV infection can be cut down by use of condoms and spermicide.

• The WellCare Lifestyle Clinics are on: 0808 1003133.