Michael Pollan has written for many years, brilliantly, about our relationship with food and farming, in particular for the New York Times. In 2018, in what seemed like a midlife departure, he published a book on “the new science of psychedelics”, which was a personal report on renewed scientific interest in experiments with LSD and Ayahuasca, after decades of taboo. Pollan saw no change of direction in that project, however; he insisted to me at the time that it was simply a natural evolution of his “abiding interest in how we interact with other plant and animal species and how they get ahead in nature by gratifying our desires”. The desire to change consciousness was a fundamental element of that relationship, he suggested. This book, which concerns our species’ symbiotic entanglements with three other potent plant-derived substances – opium, caffeine and mescaline – is a further development of a lifelong inquiry, which began, he writes, when he took up gardening as a teenager and attempted to grow cannabis.

His essays on perhaps the three most dramatically efficacious medicinal compounds proceed in a similar way, weaving personal experimentation with each of the “drugs” into informed histories of the ways in which they have taken such a hold of different human cultures. At the root of each case study is a pair of questions: the first asks why, as a species, we have gone to extraordinary lengths to propagate and disseminate these consciousness-changing molecules, and the second is why they are subject to paranoia and regulation in differing degrees.

Pollan’s personal history with opium began 25 years ago, when Harper’s magazine commissioned him to grow opium poppies in his garden, and to report on the ethics and effects of his harvest. What began as a sort of academic prank at the height of America’s “war on drugs” quickly became something a little more alarming, as Pollan realised that if purchasing poppy seeds from a garden catalogue was in itself innocent, it became immediately illicit if they were grown in knowledge of their opium-producing qualities. When a lawyer read his article pre-publication it seemed he might be risking 20 years in jail and a $1m fine. Some swift edits were made, and the full piece is published here in its entirety for the first time. There is, as Pollan observes, a profound irony in the fact that at the same moment as his quaint domestic efforts to make poppy tea were being outlawed, the Purdue corporation was patenting the slow-release opiate painkiller OxyContin, the drug behind the “opioid crisis” that has killed at least 230,000 people, and made addicts of millions more.

Far less fraught, but no less eye-opening, are Pollan’s adventures with caffeine. He deftly deconstructs the coffee and tea drinking rituals that are enmeshed in billions of lives, the social habits that have formed around our brain’s subconscious cravings for caffeine (or 1,3,7-trimethylxanthine). Pollan’s wife is far from alone in thinking of her morning coffee hit as a “cup of optimism”; the great naturalist Alexander von Humboldt described his as “concentrated sunshine”. Those compulsive associations extend to other corners of the natural world. Some plants, Pollan discovers, offer bees a small shot of caffeine as they collect nectar, an evolutionary trick that serves to sharpen a bee’s focus, and make it an even more efficient pollinator.

While Pollan goes cold turkey from his own caffeine habit – a morning trip to the local coffee shop, followed by a pot of green tea and maybe an afternoon cappuccino – he attempts to summon enough writerly concentration to detail the ways in which coffee houses and tea drinking successively fuelled the Enlightenment and the Industrial Revolution. By the mid-19th century, he reports, caffeine had become so material to creative production that Balzac used to imbibe coffee grounds dry, on an empty stomach, in order, he wrote, to “quick march ideas into motion” as “the cavalry of metaphor deploys with a magnificent gallop”. Pollan’s period of espresso-denial suggests similar conclusions – “What work of genius has ever been composed on chamomile tea?” he wonders – though he does find himself suddenly sleeping like a teenager.

The third strand of his inquiry returns him to the paradigm of all such accounts, Aldous Huxley’s experiments with mescaline that produced his 1954 book, The Doors of Perception. Pollan effectively updates that experience in light of all that has followed, for a generation more in tune with The Wire and Breaking Bad than the Grateful Dead and Timothy Leary. He “peeps inside the tepee” of peyote ceremonies that have been a foundation of Native American cultures for at least 6,000 years, and cooks up his own cacti in search of afternoon nirvana. As with all the explorations in this spirited and informed series of quests, the results of these experiments open up as many public questions as private epiphanies. Pollan is the perfect guide through this sometimes controversial territory; curious, careful and, as his book progresses, increasingly open minded.

• This Is Your Mind on Plants by Michael Pollan is published by Allen Lane (£20). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply