Barts pathology museum is usually open to the public only by special appointment. But today, I’m in luck. I find myself with an unexpected open sesame in the form of Dr Jonathan Kennedy, the director of the MSc and iBSc global public health programmes at Barts and the London Medical School, and while he has his photograph taken up on one of its mezzanine floors, I’m free to wander around alone. (At least, I think I’m alone; the museum is nothing if not ghostly.)

On the same site as the hospital in the City of London, this purpose-built, glass-roofed Victorian building is home to about 4,000 medical specimens, the majority of them displayed on open shelves. Every part of the body is represented and every kind of illness and injury – though tracking down a particular condition can be tricky for the non-medical. When I finally find the gnarled, yellow spine of a patient who suffered from tuberculosis – I was after something that speaks to Kennedy’s new book, which is about infectious disease and its effect on human civilisation – it’s largely down to luck. The vertebrae in question just happen to be next to the museum’s most famous exhibit: the skull of John Bellingham, who assassinated the British prime minister, Spencer Perceval, in 1812.

In Pathogenesis: How Germs Made History, there is no mention of Perceval. But Bellingham’s skull, preserved after the dissection of his body that followed his hanging, inevitably brings to mind Kennedy’s chapter on the Industrial Revolution – a period when previously uncommon pathogens such as TB and cholera began to thrive in our cities (cholera did not appear in Europe until the 1830s, having hitched a lift in both the guts and the linen of travellers from south-east Asia). Not only did infectious diseases then account for 55% of deaths in London; cholera epidemics in particular also led to a rise in the public’s distrust of doctors, who were sometimes attacked as they transported suspected sufferers to hospital for isolation.

Murderers’ bodies were sent for dissection by law, as Bellingham’s had been; the medical profession was hungry for cadavers and this ensured a supply line. However, as Kennedy notes, the 1828 murders committed by William Burke and William Hare, who sold their victims’ bodies to Edinburgh University’s medical school, caused people to believe that cholera patients, removed from their communities by doctors, were about to be similarly dispatched. The “anatomists”, of course, were absolutely right to compel isolation, but most of their patients would indeed die, even if they didn’t kill them personally. So far as treatments went, doctors were clueless. One of their more bizarre cures involved injecting turpentine and mutton stew into the patient’s intestines via the anus.

Kennedy’s book, which aims to show how infectious disease has shaped us from the time of the Neanderthals to the era of Covid-19, is full of amazing facts like these. For days after I read it, I kept thinking of Pope Clement VI who, when the Black Death reached 14th-century Avignon (where he was exiled), retired on the advice of his physician to his chamber, seeing nobody, and spending all day and all night between two great fires – the flames of which kept the rats that were carrying the disease at bay. Clement did very well to survive. It’s estimated that somewhere in the region of 60% of the population of Europe (about 50 million out of 80 million people) died from the plague between 1346 and 1353. Meanwhile, the economy more or less ground to a halt.

Kennedy is a sociologist who somehow ended up – he describes “a meandering intellectual journey” – in public health. “Before coronavirus, I published a paper looking at the links between populism and vaccine confidence,” he says. “So when the outbreak started, the rational thing to do as an academic might have been to get my hands dirty working on Covid-19. But actually, I struggled with the way my work was suddenly the news and so I went in a slightly different direction.

“With Pathogenesis, I wanted to broaden [the discussion] out. In the pandemic, most experts stopped saying ‘this is an aberration’ pretty early on, for the simple reason that if you look back, you can see that it really isn’t exceptional. Over the course of human history, it has been normal to have these pandemics that not only kill a lot of people and bring civilisations crashing down, but which also, sometimes, create space for new kinds of societies to emerge. The book is an intellectual response to the slightly myopic way some people viewed the pandemic. It’s a long view, one that we couldn’t perhaps take at the time because everyone was in such a panic.”

Pathogenesis doesn’t only cover thousands of years of history – it seeks radically to alter the way the reader views many of the (often very well-known) events it describes. Having read recent research by scientists, archaeologists and anthropologists, for instance, Kennedy suggests that Homo sapiens became the dominant species of human not as a result of any inherent intellectual superiority over Neanderthals, but because our immune system was better suited to adapting to pathogens as we migrated north from Africa. He also looks at the role disease played in colonisers’ ability (or not) to govern their empires.

It’s generally accepted that Rome fell partly because of plague (probably smallpox), its armies weakened by illness. But by his telling, had it not been for an Ebola-like pandemic in the second century, Christianity would never have become so widespread – a slightly more surprising analysis. What part, he asks, did pathogens play in the near-miraculous victories of conquistadors such as Hernán Cortés and Francisco Pizarro in Central and South America in the early 16th century? Kennedy, as you will have gathered, doesn’t believe in the so-called great man theory of history. He believes in the sick man theory of history, for no soldier can march if he is dying and no king can rule who is weakened by infection.

With this book, he set himself a mammoth task. Where on earth did he start? “Well, I guess my starting point was microbes,” he tells me (we’re in a seminar room now, far away from all the pickled lungs and hearts). “Within us, there are 40 trillion bacteria. They’ve evolved over millions of years and we now know they’re playing all sorts of roles in our lives. This blows my mind and the question I wanted to ask as a sociologist was: if these tiny things have such an influence on our bodies, what kind of effect might they have on the body politic, the body social, the body economic? Humans make history, but not in circumstances of their own choosing. Microbes are a huge force in creating those circumstances.”

Together, we roam haphazardly over some of the history in his book. I want to know what he found most mind-boggling. “One thing that really came out of the whole project was the absolute baselessness of white ideology,” he says. “Obviously, that comes to the fore when we’re discussing colonialism. It seems bizarre to me now that people would put down Pizarro’s conquering of the Inca empire with only 168 troops to the fact that he had superior weapons. It’s hard to overstate the role that infectious diseases played in the foundation of the Spanish empire.” The Spanish brought smallpox with them, among other diseases, which devastated the Indigenous population. This was followed by measles, an infection now thought to have been similar to salmonella, and flu. Local populations fell by 90%. The Spanish invaders interpreted all this as a sign that God supported them.

Yet we cling to assumptions about superiority. “Ever since we discovered the Neanderthals were a separate species, we have dehumanised them, seeing them as these brutish cave men,” Kennedy goes on. “But it seems like they were just as smart as us. They could talk, they created cave art. They buried their dead and observed rituals. To realise that Homo sapiens were migrating out of a tropical region, and that they were carrying more microbes, and that the Neanderthals further north struggled to deal with those… it’s a neat reflection of the arguments about colonialism.”

In the context of our own times, Neanderthal man is particularly interesting to him. We now know that Homo sapiens and Neanderthals interbred, with the result that we carry their genes. The Neanderthal genome has, moreover, been sequenced (by Prof Svante Pääbo, the winner of the 2022 Nobel prize for physiology), and papers published since have shown that Neanderthal gene variants in the contemporary population have had an impact on responses to Covid (those carrying one such variant were 22% less likely to end up in intensive care compared with infected people of the same age). “It’s crazy to think that something that happened 60,000 years ago impacts on whether or not we have a really terrible reaction to Covid,” says Kennedy.

But if he is fascinated by the past, he’s also concerned about the future. “We’re living in a golden age for microbes. Population densities are increasing, people are moving more quickly around the world, the climate is changing. We’ve seen the emergence not just of Covid, but of HIV/Aids, Zika, Dengue fever, Sars and Ebola. It seems now that we won’t be able to conquer infectious diseases. Rather, we’re going to have to learn to deal with the new diseases that periodically arrive to threaten us.” If history teaches us anything, he believes, it’s that things will only change in terms of health if we stick together: “These are global societal issues. If one rich country tries to save itself, it’ll still be doomed.”

Has writing Pathogenesis made him feel pessimistic? On balance, it hasn’t. While antibiotics may soon cease to be effective in treating some illnesses, a prospect he finds frightening and “hard to fathom”, in the future microbes may themselves provide the solution: “Viruses might be the new antibiotics. They kill about half the bacteria in the world every day. It’s about finding the right one to target the right bacteria.”

He thinks of Antonie van Leeuwenhoek, the 17th-century Delft haberdasher who pioneered the microscope. “No more pleasant sight has ever come before my eyes,” wrote Van Leeuwenhoek, enraptured by the teeming new world he saw beneath his lens. There is, Kennedy says, still so much to discover. “We’re incredibly vulnerable as a species, but we’re also incredibly ingenious. A hundred and fifty years ago, we didn’t even know that microbes cause infectious diseases. Now we’re analysing Neanderthal DNA.”

• This article was amended on 2 April 2023 to correct a mistranscription; Dr Jonathan Kennedy spoke about the interbreeding of Homo sapiens and Neanderthals as “something that happened 60,000 years ago”, not 6,000 years ago as an earlier version said.

• Pathogenesis: How Germs Made History is published by Transworld (£25) in the UK and Torva ($35) in Australia. To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply