"One of our first patients, just 17 years old, was brought to us in a wheelchair," says Professor Christer Lindquist, a pioneer in the use of a brain surgery technique for people with Obsessive Compulsive Disorder (OCD), known as Gamma Knife. "This boy would set himself maths problems, which he had to solve before he could eat. His OCD had become so severe, and the maths problems he set himself so complex, that he couldn't solve them any more, so he couldn't eat."

At Butler hospital in Providence, Rhode Island, Lindquist and colleagues put the boy in an MRI-like machine and passed beams of gamma radiation through his brain. These beams converged on a pinpoint-accurate spot where they created a lesion that damaged a tiny area of tissue, blocking the pathway that caused the OCD symptoms.

This is modern psychosurgery, a hi-tech, experimental, descendant of the now infamous frontal lobotomy. It could offer hope to millions suffering from OCD, and other disorders such as severe depression.

Over the past 10 years, Gamma Knife has become a highly effective treatment for brain tumours and there are now several Gamma Knife centres in the UK. Nick Plowman, consultant clinical oncologist at St Bartholomew's hospital, London, says that before Gamma Knife, an acoustic neuroma– a benign brain tumour – would require a major operation. "Now you can do it in a fraction of the time, without opening the head, whilst the patient listens to music." Successful for nine out of 10 patients with this type of brain tumour, the radiation stops the tumour cells from reproducing, and in time they'll die.

The surgery is also now widely used to treat certain brain conditions, such as a rare form of epilepsy and a condition called trigeminal neuralgia, where the patient experiences shooting pains in the face. Still, it could be a while before OCD sufferers will be offered Gamma Knife surgery in the UK. For this and other psychological problems, such as depression, Gamma Knife is still considered by many to be highly controversial.

OCD affects around 2-3% of the UK population, and the usual treatments are medication or Cognitive Behavioural Therapy (CBT). But these don't work for everyone. Joel Rose, director of the UK charity OCD Action, is not surprised that some are prepared to try experimental brain surgery. "People become paralysed," he says. "They're in a despairing state and they'll try anything to get out of it."

However, many surgeons believe that we don't know enough about the brain circuits to tamper with them. "When it comes to treating OCD and other psychological disorders, Gamma Knife is totally unproven," says Plowman. It is certainly in its infancy. Lindquist carried out the first Gamma Knife treatments for OCD at Butler/Rhode Island in 1992. Since then, 56 people have had the procedure.

About 60% of the patients at Rhode Island were much improved, but many were left with residual symptoms. "This might all sound lame," says Lindquist, "But you have to bear in mind that these people are suffering severely. They've been treated for years with the most advanced drugs and CBT."

It can take up to a year to see any improvement, and "even if neurosurgical intervention is successful," says Richard Marsland, a psychologist at Butler who helps screen the patients, "they have to be included in an aftercare programme. Most patients acquired their OCD at an early age, and missed at least part of their normal development . . . they have to catch up."

Gerry Radano, a former flight attendant from New York State, is one of the most vocal supporters of Gamma Knife for OCD. She was in three psychiatric hospitals and tried every medication available before having the surgery. "Gamma Knife is the best thing that ever happened to my OCD life," she writes on her website.

But there are scare stories. One Ohio hospital stopped performing the procedure after a law suit in 2002, when a patient was left partially paralysed. "This should never happen," says Lindquist. In fact, he says "the main risks for the surgery are temporary lethargy or loss of initiative if too high a dose is given, which happens in around 10% of cases".

The technique certainly could not be further from the brutal lobotomies made famous by Ken Kesey's novel, One Flew Over the Cuckoo's Nest. While the frontal lobotomy essentially destroys part of the brain, Gamma Knife is highly accurate and non-invasive, damaging only a minute area - 100 millimetres square - of brain tissue. It is usually done as an out-patient procedure. Some might experience a mild headache afterwards, but most report no physical problems at all.

"One Flew Over the Cuckoo's Nest did psychosurgery no favours," says Lindquist. "The treatment of psychiatric disorders is still surrounded by an aura of mysticism. We think about the psyche as something magical, and aren't willing to accept that a psychological disease could be a transmitter problem, just like Parkinson's disease." However, for OCD he admits, "it's still controversial. We have to be extremely careful that patients have exhausted all other avenues." As for the 17-year-old boy Lindquist operated on years ago? He is now a doctor himself.

• This article was amended on Tuesday 15 December 2009 to clarify the size of the area of the brain affected by the Gamma Knife surgery.