The love of islands is a widespread affliction – why else are we still reading Robinson Crusoe after 300 years? Why Treasure Island? Why after 75 years and over 2,000 episodes are we still listening to Desert Island Discs? From the blessed isles of Tír na nÓg and Thomas More’s Utopia to the island-dramas of CS Lewis and Enid Blyton, it seems we can’t get enough of them.

My own island journeys began in the 1980s, as a boy in my local library in Fife. While my mother browsed the shelves, I’d oftensit down on the scratchy carpet tiles and open an immense atlas. Galaxies of islands were laid out for exploring across the Pacific, the Mediterranean, the Caribbean, and I’d run my fingers over each archipelago as if reading braille. I hardly dared hope I’d reach any of them.

I almost became a geographer but went instead to medical school in Edinburgh. But on finding myself with a few days off, I’d leave the buzz of the city behind to hitch north or west, and get myself on to a ferry. It wasn’t that university life was unsatisfying, or that I was trying to “get away from it all”; instead, I felt that island travel offered the chance to gather perspective – to feel part of a world somewhat emptied of the human, to recalibrate.

There was a creative tension between the extremes of island and city that I began to enjoy exploring. I remember hitchhiking one year to Unst in Shetland, to reach the bluff overlooking the Muckle Flugga gannet colony – almost the most northerly point in the British Isles. There was rumoured to be an albatross – a bird of the southern hemisphere – living there among the gannets, and I was enthralled by the possibility of seeing this visitor from some of the remotest seas on the planet. I passed a winter solstice on North Ronaldsay in Orkney, explored the west coast of Greenland, caught ferries between islands in the Aegean and down the west coast of Chile. It felt that with each of these journeys I was in pursuit of something.

Why is it that so many seek out islands for an imagined peace or tranquillity? The psychoanalyst Donald Winnicott had a great deal to say about the therapeutic value of isolation. Speaking in particular of adolescence, he made a distinction between “isolation”, which could be helpful, and “insulation”, which was pathological. He thought that we need to access isolation in order to develop a sense of wellbeing, independent from that of relatives or therapists. But at the same time we have to guard against becoming “insulated”, by which he meant impervious, closed off and unreceptive to whatever life is trying to teach us. It’s an increasingly difficult balance now that many of us – not just adolescents – are coping with a surfeit of connectedness. As online connectivity has proliferated so has a creeping epidemic of anxiety. It’s more vital than ever to find new ways to disconnect.

Through my 20s and 30s a pattern of life became established: periods of intense, vivid connection in cities through medicine, followed by periods of silence, retreat and isolation; as the medical work became more intense and demanding, so did my periods of isolation. One year I left a busy job in emergency medicine to go to live in Antarctica for 14 months. After another challenging surgical job I signed up as a warden on the Isle of May, a bird reserve off the coast of Fife where the work was outdoors, silent, refreshingly physical – so different from the brain-work and clamour of A&E. Another year I pulled myself from the wreckage of a disintegrating love affair to reach Mount Athos, a Greek peninsula (and island in all but name). Orthodox monasteries on Athos have been offering hospitality to visitors and pilgrims for a thousand years, and as I hiked clockwise between them the glitter on the sea, the silence of the forests began to still the agitation in my mind.

In the opposing allures of island and city there’s a paradox at work: that in the city it’s easy to become isolated and lonely, and on an island to feel part of a community. It’s as if with the former’s near-infinite possibilities of connection we can’t help but let most of it slip through our fingers: you can’t connect with everyone, and so you end up connecting with no one. And the converse is often true of small island communities – something I’ve seen first-hand as an island GP, where connections often run deep, and there’s a complex network of mutual reliance that seems enviable from a city perspective. On a small island there are challenges of resources, of transport, of weather, but often those shared challenges help bring a community together, rather than drive it apart.

I’ve always loved the passage in Herman Melville’s Moby-Dick when the Pacific Islander and harpoonist Queequeg is described as “entirely at his ease; preserving the utmost serenity; content with his own companionship, always equal to himself”. While in Antarctica I read up on the psychology of what are still known as “isolated and confined environments” – deep-sea trawlers, polar bases, space stations. Many who’ve thrived in such environments have come to learn something about themselves that has made them happier, more resilient humans. This year, we’ve all learned something through the isolations of the pandemic.

There is a risk here of romanticism, when the reality of island life is very different from the one imagined by city-dwellers. And it’s also true that one man or woman’s island can be another’s metropolis – for the Faroese, Orcadians are well-connected, while the islanders of Tristan da Cunha envy those of St Helena (the latter have an airstrip, the former only a harbour).

There’s no firm measure of what constitutes isolation – we are in the realm of ideas. Between island and city, romanticism and reality, there’s a balance to be uncovered. It’s a balance that I’ve spent a lifetime exploring and I’m happy to carry on with my search.

Island Dreams by Gavin Francis (Canongate, £20) is out on 1 October. Order it for £17.40 from guardianbookshop.com