Jose Montoya was a trainee doctor when his supervisor told him that if he continued specialising in treating chronic fatigue syndrome (CFS), he would end up homeless. “About 15 years ago, I started working with 10 patients who’d had their lives devastated by this illness,” says Montoya, now a professor at Stanford University and one of the world’s leading experts on the disease. “I had been able to help them, so I took my results to my academic mentor and he told me: ‘You are committing academic suicide. You’re turning your career into a mess.’”

Months later, Montoya, was travelling to a conference in Paris with his mentor when the subject came up again. “There was a homeless man lying drunk in the street, and he pointed to him and told me: ‘That’s how you’re going to end up if you keep studying chronic fatigue syndrome.’”

CFS is believed to affect around 1 million Americans and approximately 250,000 people in the UK. The effects can be devastating. “My son has CFS and is only able to get out of bed for half an hour a day,” says Mary Dimmock, a CFS activist who worked in the pharmaceutical industry for 30 years. “The disease isn’t so much about tiredness, it’s that the patients just collapse if they go beyond their available energy. These people are so sick that they may only have enough energy to brush their teeth or chew their food.”

Yet for much of the past three decades, CFS has been treated as the proverbial skeleton in the closet of the medical world. Potential researchers have been scared off by the stigma associated with the disease, and government funding has been nonexistent. “When I was a medical student in the 90s, we were instructed that CFS patients could not be seen in our clinic,” Montoya recalls. “And a letter was sent out to those patients telling them not to come.”

To understand why the medical industry has treated the illness with such disdain, you have to go back to 1955 and a sudden outbreak of CFS at the Royal Free Hospital in London which affected approximately 300 people and led to the closure of the hospital for three weeks. The causes of this mystifying epidemic were unknown, but pathology findings suggested something had triggered inflammation in the brain and spinal cord, but with no obvious cause, health officials charged with investigating the outbreak two decades later concluded it had been down to mass hysteria. In the 1980s, psychiatrists in the US and UK involved in investigating a similar CFS epidemic in Nevada decided the illness was largely psychogenic, a result of patients believing they were really ill and allowing themselves to become deconditioned. It’s a tag that has stuck to this day. In 2011, the Pace trial – a five-year study of CFS funded by the UK government, recommended cognitive behavioural therapy and graded exercise regimes as treatments for the disease.

Over the past 20 years, though, a handful of scientists have defied convention by looking deeper into the disease than ever before, sometimes inspired by chance events. Professor Garth Nicolson, founder of the Institute for Molecular Medicine in California, noticed a wave of CFS in soldiers returning from the 1990-91 Gulf war, among them his own daughter. “The more we looked into it, the more we found that infections appeared to be the root cause, which was why some of the sufferers transmitted CFS to family members,” he says. “Infections aren’t a universal cause, but they are definitely one of the main contributors.”

Nicolson and others believe that as with so many complex diseases, CFS is really an umbrella term for a whole variety of underlying illnesses – some psychological or due to anxiety-related exercise intolerance, but many others down to neurological impairment, energy-production impairment or even autoimmune dysfunction. “We’ve been taking diseases that aren’t even related to each other, but because fatigue is the unifying symptom, we’ve thrown them in the same bucket and treated them the same,” Dimmock says. “If you prescribe a graded exercise regime to a CFS patient with an energy-production impairment, you can do a lot of harm. It’s like taking patients who all suffer from shortness of breath, but the root causes range from asthma to angina. You would never treat them as the same group of patients.”

Montoya believes that around 80% of the CFS patients he treats have developed their condition as a result of infection. But because few of them get to see a specialist until they have been ill for many months or even years, the bacteria or viruses responsible have long gone into hiding inside the body’s cells, meaning that many standard blood tests appear to show there is nothing wrong. “This is why so many doctors have dismissed CFS as psychological in the past,” he says.

At the Mailman School of Public Health at Columbia University, Professor Mady Hornig studies the various causes and potential treatments for CFS. “The range of symptoms across CFS patients is extremely diverse,” she says. “In some cases, something may have damaged the mitochondria which provide energy for immune cells, brain cells and your muscle cells.” Other researchers in the US have suggested that such patients can improve a lot from treatments such as a membrane lipid replacement. “For others, viruses or bacteria appear to have induced antibodies that can react against parts of your bodily organs, including your brain and muscle tissue, causing disruption,” Hornig says. “There’s a group in Norway who have had success using immunotherapy to treat some CFS patients.”

But while these scientists have been fighting a lone battle for many years, the tides of change seem to be finally sweeping through the medical community. Last year, the Institute of Medicine in the US issued a report describing CFS as a “serious, chronic, complex and systemic disease”. Many hope this will lead to new investment and new treatments. “Pharmaceutical companies have not been able to invest in the disease in the past as there’s been such little funding for academic research,” Dimmock says. “The companies need that knowledge to try to develop medicines.”

Montoya and Hornig have both received grants from the National Institute of Health to identify the subgroups of patients within CFS more accurately. “We need to try to develop biomarkers for CFS that pinpoint what the root causes are,” Hornig says. “For some people it may be abnormal gut bacteria which can damage immune system function, and so probiotic treatment may help. Others may have more profound brain fog and sleep disturbances which could be due to an infection moving into their spinal fluid and brain, so they may respond preferentially to antibiotic or antiviral treatment.”

But, ultimately, the holy grail for all CFS researchers is to understand why some people are susceptible to the illness while others may be exposed to the same infections or causes, and yet experience little change. “That really is the big question,” Montoya says. “If you take Epstein-Barr virus, which causes glandular fever, only about 10-15% of people tend to go on to have a prolonged case of the disease leading to CFS. But why do this set of individuals have this type of response to certain viruses while others don’t? Are there genetic factors, is it actually a completely different pathogen which we haven’t discovered yet? We don’t know but with more financial resources being committed and institutes across the world concluding this is a disease which requires serious research and attention, I believe we will know the answer within the next five to 10 years.”

• This article was amended on 5 April 2016 to correct two quotes from Professor Mady Hornig and clarify her affiliation.