In 1989, Charles Janeway, a scientist at Yale University, had an epiphany that would revolutionise immunology. For 50 years, immunologists had subscribed to the dogma that vaccines worked by training the body to recognise molecules that were foreign to the body – “non-self” in immunological jargon. The usual way of doing this was to use vaccines to expose people to a dead or harmless version of a microbe, prompting the activation of antibodies that would be ready to swamp the germ should they encounter the alien entity a second time.

But there were exceptions to the rule: sometimes, proteins separated from originating germs proved ineffective as vaccines; at other times, vaccines required the addition of an adjuvant, such as aluminium, to kickstart an immune response and no one could explain why. What if, wondered Janeway, the presence of something that had never been in your body before was not sufficient to trigger an immune reaction? What if a second signal was required?

Today, that second something is known as a pattern-recognition receptor and it is understood that there are countless varieties of them, each equipped to detect specific types of germs and switch on the appropriate immune responses. Together with an alphabet soup of other specialised cells, hormones and proteins, they form part of our innate immune system, helping us to distinguish harmful bacteria and viruses from beneficial ones, such as gut microbes essential for digestion.

For Daniel Davis, professor of immunology at the University of Manchester, they constitute a “beautiful cure” more powerful than any product of a pharmaceutical laboratory. Yet it is only in the past 30 years that immunologists such as Davis and Janeway, who died in 2003, have begun to shed light on these “wonders taking place beneath the skin”.

In the process, they have found new ways to treat cancer, diabetes, arthritis and other age-related diseases. Immunologists are even beginning to understand the way in which immune responses are dependent on emotional and psychological states and the role that stress and exposure to light play in fighting disease.

Given this, you would have thought that research into the workings of the immune system would be a top scientific priority. But while billions have been poured into the pursuit of the Higgs boson, immunology lacks a similar programmatic call-to-arms. Instead, Davis argues, immunology has always been a curiosity-driven science, a matter of “a few individuals following their nose”.

This is nowhere more true than in the case of interferon. A signalling protein involved in a host of immune responses, interferon owes its discovery to a chance meeting in 1956 between two scientists at the National Institute of Medical Research in Mill Hill, north London. At a time when their colleagues were focused on the epidemiology of flu, Alick Isaacs and Jean Lindenmann asked a completely different question: namely, why was it so rare for someone to be infected with two different viruses at the same time? That observation went back at least as far as Charles Darwin’s grandfather, Erasmus Darwin, who commented he had never seen a patient with measles who also had smallpox, but until Isaacs and Lindenmann no one had thought to investigate the phenomenon.

They found that by signalling genes to produce proteins such as tetherin to attack viruses, interferon played a crucial role in this process. In an example of how in science everything comes full circle, recent studies even suggest interferon may help people stave off flu, explaining why people who lack a key interferon-stimulating gene are more likely to be admitted to hospital for the disease. This is important because if people who lack the interferon gene could be screened and prioritised for vaccination in the autumn, it could prevent hospitals being swamped by elderly patients in winter.

Davis is a sure and engaging guide to these developments. Beginning with Janeway’s prediction of pattern-recognition receptors, each chapter is devoted to a scientist, or often a pair of scientists, who, working outside the mainstream, thought to ask questions no one else was asking at the time and often had to endure years of scepticism and scorn before seeing their ideas accepted (unfortunately in Janeway’s case he died before the award of the Nobel prize to a colleague whose research was inspired by his theories).

These mavericks include Ralph Steinman, the Canadian immunologist credited with the discovery of dendritic cells, and Jim Allison, whose discovery of “immune checkpoint therapy” is fast becoming an important adjunct to radiotherapy and chemotherapy for the treatment of cancers. In each case, Davis shows how these scientific thinkers overturned the previous dogma and progressively deepened the story of immunology.

His message is that although knowledge of the immune system has come on in leaps and bounds in the past 100 years, immunology still lacks a unifying theory. “We must not expect everything the immune system does to fit any one over-arching principle,” he concludes. “The system discriminates between self and non-self, and it detects germs, and it responds to danger, and it does all these things concurrently – and messily.”

• The Beautiful Cure by Daniel M Davis is published by Bodley Head (£20). To order a copy for £17 go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99