The poet Edmund Blunden wrote in 1942: "We have been increasingly on pilgrimage." We are once again increasingly on pilgrimage. A revival is under way worldwide, with pilgrim numbers rising even as church-going figures fall. Medieval hostelries on the roads to Santiago de Compostela, closed for centuries, are reopening to cater to the volume of travellers. In 1985, 2,491 people received the certificate of completion known as la autentica; more than 270,000 did so in 2010. On the little north Norfolk village of Walsingham – site of an 11th-century vision of the Virgin Mary, recently self-branded as "England's Nazareth" – a quarter of a million pilgrims now converge each year, including participants in the "children's pilgrimage", the "youth pilgrimage" and the "Tamil pilgrimage". Last year's hajj to Mecca was the most populous ever. At the 2010 Kumbh Mela, the Indian space agency took satellite pictures of the tens of millions of pilgrims in order to improve the government's crowd control at future melas. Rowan Williams spoke recently of "a whole generation of new pilgrims … wishing to cut through the clutter of institutions, and achieve self-discovery in a new place".

Across faiths and denominations, down the green lanes of England, along the dusty roads of Spain, up the cobbled streets of Alpine towns, through the marl deserts of Israel and the West Bank, around the sacred peaks of the Himalayas, over the frozen lakes of Russia and along the holy rivers of India, millions of pilgrims are on the move: bearing crosses, palm branches, flaming torches, flower garlands, prayer flags and over-stuffed rucksacks, clutching scuffed wooden staffs or shiny trekking poles, and tramping, prostrating, hobbling, begging and believing their ways onwards, travelling by aeroplane, car, bus, horseback and bicycle, but most often on foot and over considerable distances – for physical hardship remains a definitive aspect of most pilgrimage: arduous passage through the outer landscape prompting subtle exploration of the inner.

This pilgrimage revival is not only religious in nature; it also extends widely and fascinatingly into secular culture and art. Writing Britain, this summer's British Library exhibition about landscape and literature, teems with pilgrims and their peregrinations, from Chaucer's convivial chevaliers onwards. Chaucer began the first of his Canterbury tales 625 years ago: to mark the anniversary, a group recently re-walked the route from the Tabard Inn in Southwark to Canterbury Cathedral in Kent. They stayed in the original stopover towns, read and re-enacted the tales along the way, and recorded their journey in multimedia form as it unfolded. Later this month a weekend-long arts festival dedicated to pilgrimage will take place at Mount Amelia in Norfolk, involving poetry, opera, film, performance, photography, sculpture, improvised jazz and storytelling.

Certainly the most influential work of travel writing published in the past 20 years is WG Sebald's The Rings of Saturn, whose original German subtitle was Eine englische Wallfahrt – "An English Pilgrimage". Told in winding and digressive sentences, and studded with captionless black-and-white photographs, the book recounts a foot-journey taken by a narrator who closely resembles (but is not quite) Sebald himself, down the crumbling coastline of Suffolk in the "dog days" of summer. Place, in Sebald's pilgrimage, presses hard upon the walker: the paranoid East Anglian coast, fortified for thousands of years against various kinds of invasion (sea-walls, Martello towers, pill-boxes, radar stations, cold war listening posts), induces first joy, then fascination, then neurosis and at last breakdown.

The Rings of Saturn has provoked its own cult, and its own metapilgrimages. This year saw the international premiere and success of Patience (After Sebald), Grant Gee's feature film tracing the book's itinerary and its stories. I walked the route myself over several days several summers ago, when starting to write an oblique biography of Sebald. My hope was that by means of footstepping his shade, I would understand the man and his work better. But the sun was hot, children were having fun in the fountains at Lowestoft, and in the end I gave myself up to the pleasures of the walk. Somewhere near Benacre Broad I plunged into the warm waves to wash off the grayscale of the Sebaldian worldview; a few months later I abandoned the book project entirely (and straightaway felt better about life).

Last year Colin Thubron published To a Mountain in Tibet, an austere and moving account of his journey to Kailash in western Tibet, the holiest peak of the Himalayas, where the Ganges, the Brahmaputra, the Indus and the Sutlej all have their source, and around whose base pilgrims have long made circuits of notorious difficulty. The most extreme form of this kora, or circumambulation, requires body-length prostrations for 32 miles of rocky path, at up to 18,000ft in altitude. Thubron made his own secular pilgrimage in memory of his recently deceased mother, and he circled the mountain on foot with Hindus, Buddhists, Jains and Bonpos as his chance companions. Incomprehension and cold comfort were the chief yields of his journey, though, and the book hums with strange loneliness and grief – a different dark pilgrimage to set alongside Sebald's.

Also last year the writer-artists Alec Finlay and Ken Cockburn completed The Road North, a project inspired by the 17th-century Japanese pilgrim-poet Matsuo Bashō, who perfected the haibun form of prose-poetry as a means of recording the thousands of miles he wandered through Japan. Guided by the example of Bashō, Finlay and Cockburn followed their own narrow roads through Scotland, stopping at 53 "stations" and creating a vast "word-map" of the country, made up of prose-poems, renga, hokku and blog posts. Another Scottish work, Nan Shepherd's delicate meditation on place and psyche, The Living Mountain, is finding a new readership, 60 years after it was written and 35 years after it was first published. "I believe that I now understand," concludes Shepherd, "why the Buddhist goes on pilgrimage to a mountain. The journey is itself part of the technique by which the god is sought."

The artist Richard Long continues to make his epic walks, distinguished by their rites of way (a flint carried from one coast to another, stones or sticks left in patterns by the roadside), and pilgrimage remains vital to the practice of other influential sculptors and artists: Hamish Fulton, for instance, whose book Wild Life records the hundred days of ritual walks he has undertaken in the Cairngorm massif (a circumambulation of Ben Macdui on the night of the summer solstice; a seven-day walk conducted in silence; a pacing-off of the shoreline of high and lucid Loch Etchachan), or Chris Drury, whose work has long been preoccupied with cairns and waymarkers, or Gary Fabian Cooper, whose recently published Dartmoor Home details his chronic commitment to re-walking the same local paths and routes – pilgrimage within perimeter.

Next month sees the premiere in Manchester of Swandown, a film by Andrew Kötting about a "water pilgrimage" he made with Iain Sinclair, sailing (well, pedalling) a swan-shaped white pedalo from Hastings on the English Channel to the Olympic site in East London. Part send-up of the ODA's commitment to "diversity" and "ambition", and part playful picaresque, the film records the journey and its meetings. Often funny, sometimes poignant, and well aware of its own absurdities, the film shares its charms with early physical comedy cinema: Buster Keaton at sea, perhaps. In method at least, Swandown recalls another of Sinclair's famous ritual journeys, his walk along (around) the M25: a 117-mile stalk of the city's greybelt, which attempted both to "exorcise the unthinking malignancy" of Blairism and "to celebrate the sprawl of London".

As Sinclair and Kötting well know, the pilgrim always treads the brink of self-parody, and is often closely followed by heritage and by hokum. Pilgrimage has gone cult – it has also gone kitsch. TC Boyle's pin-sharp parody of American nature writing, The Tortilla Curtain (1995), features a Californian essayist called Delaney Mossbacher, "a liberal humanist with an unblemished driving record", whose personalised registration-plate reads PILGRIM and who styles himself in his writing as "a seer, a worshipper at the shrine" of Mother Earth, stepping forth with "manzanita stick in hand and nylon pack clinging to my shoulders like a furled set of wings". Yes, pilgrimage is still a word that can labour under its burdens of piety. And yet it also feels like a necessary term for describing how more and more people are choosing to make sense of their places and of themselves.

Five years ago, I began a series of journeys along the old ways of the British landscape: its holloways, field-paths, Neolithic tracks, sea-roads, coffin-routes, drove-roads and pilgrim-paths. "Knowing how way leads on to way …" wrote Robert Frost in what is surely the best-known path-following poem of them all, "The Road Less Travelled". I set out walking on the chalk of Cambridgeshire, and ended up on the dolerite and gneiss bird-islands of the Outer Hebrides, the granite of the Cairngorms, the limestone of the West Bank – where a small group of Palestinians walks ancient wadi paths as protest against Israeli land control – and the moraines of Minya Konka, a sacred Tibetan peak of dazzling elegance and altitude. The Icknield Way, which runs within a few miles of my home, was my entry to a network of old routes criss-crossing Britain and its waters, and connecting them to continents beyond. Along the way – along the ways – I walked a tidal path nicknamed the Doomway, I walked stride for stride with a 5,000-year-old man near Liverpool, I traversed a stretch of the winter Ridgeway on cross-country skis with the only Marxist tax lawyer in London, possibly the world, I had a number of experiences that I still find hard to explain away rationally, and I developed some very large blisters indeed.

Everywhere I went on these journeys, I encountered men and women for whom landscape and walking were vital to life. I met tramps, trespassers, dawdlers, mourners, stravaigers, explorers, cartographers, poets, sculptors, activists, botanists, and pilgrims of many kinds. I discovered that walking is still profoundly and widely alive in the world as a more-than-functional act. I met people who walked in search of beauty, in pursuit of grace or in flight from unhappiness, who followed songlines or ley-lines; I witnessed walking as non-compliance, walking as fierce star-song, walking as elegy or therapy, walking as reconnection or remembrance, and walking to sharpen the self or to forget it entirely.

It seemed that every month I met or heard tell of someone else setting out on a walk "in search of something intangible", as Rebecca Solnit defines pilgrimage in her great history of walking, Wanderlust. A young woman, fallen under the spell of George Borrow, had tramped across England and Wales to St David's in Pembrokeshire, following only footpaths and green ways. Three folk singers called Ed, Will and Ginger had sold their possessions, left their homes and taken to the paths of England, sleeping in woods and earning their food by singing folk songs as they went. Someone was wandering the boundaries of Northamptonshire – ancestral home of Britain's boot and shoe industry – sleeping in barns and church porches along the way. One day I walked 25 miles with a young man called Bram Thomas Arnold who, following the death of his father, had set out from London to walk to St Gallen in Switzerland, where he had lived as a child. He carried his father's ashes with him, slept in a small tent by the sides of alfalfa prairies in northern France, made camp after dark and struck camp before first light to avoid farmers and police, and got as far as the Black Forest, where he caught a train the rest of the way.

Like many English walkers before me, I ended up in Spain. In Madrid I met an artist, Miguel Angel Blanco, who might have stepped from the pages of The Shadow of the Wind or Borges's Labyrinths, and who has devoted his life to the creation of an astonishing library – the Biblioteca del Bosque (Library of the Forest). His library comprises more than a thousand wooden "book-boxes", each of which is a reliquary or cabinet containing the objects and substances (snakeskin, quartz crystals, resin, elm leaf) gathered along the course of a particular walk. Each of these micro-terrains represents a completed journey; but the library itself – ever growing – is a compound pilgrimage without visible end. With Miguel as my guide, I followed a branch-line of the Camino de Santiago from Cercedilla, north of Madrid, up through the pine forests of the Guadarrama mountains, then down to the medieval city of Segovia and on out on to the scorched yellow tablelands that stretch towards Galicia and the cathedral of St James.

In the mountains of eastern Tibet, walking long miles through oak forests to reach Minya Konka, I set my pace to a Spanish palindrome on the subject of pilgrimage: La ruta nos aportó otro paso natural – "The path provides the natural next step". Its form cleverly acknowledges the transformative consequences of the pilgrimage, which turns the mind back upon itself, leaving the traveller both ostensibly unchanged and profoundly redirected.

In the Outer Hebrides, I walked from the west coast of Lewis to the south-east coast of Harris, sleeping in beehive shielings and passing under an eagle-filled sky, and met a sculptor called Steve Dilworth for whom the Harris interior – lochan, scarp, erratic boulders – had become a consecrated terrain, as mysterious and powerful in its presence as Avebury or Silbury Hill. With a Lewisian poet, sailor and storyteller named Ian Stephen, I took by boat to the North Atlantic sea-roads – the routes of long usage that exist at sea, determined by current, prevailing wind and coastline, along which people (raiders, traders, craftsmen, pilgrims) and their ideas (technologies, languages, stories) have travelled since prehistory. One week we sailed due north for 40 miles from the most northerly tip of Lewis, in an old open boat called Jubilee, to the little gannet-island of Sula Sgeir. When we reached the island we circled it by oar and by sail, and as we did so we noted off its features – Geodha a Bhuin Mhoir, Palla an Iar, Sroin na Lic – for such is the attention paid by the Gaelic language to its landscapes that even that sharp scrap of uninhabited rock, far out into the North Atlantic, has more than 30 place-names attached to it.

Not long before I went to Spain, I read an essay in the journal Artesian by a Czech writer called Vaclav Cilek, cryptically entitled "Bees of the Invisible". Cilek – himself a long-distance wanderer – proposed a series of what he called "pilgrim rules", of which the two most memorable were the "Rule of Resonance" ("A smaller place with which we resonate is more important than a place of great pilgrimage") and the "Rule of Correspondence" ("A place within a landscape corresponds to a place within the heart.") "The number of quiet pilgrims is rising," he observed. "Places are starting to move. On stones and in forests one comes across small offerings – a posy made from wheat, a feather in a bunch of heather, a circle from snail shells." I had come across such DIY land-art often myself: the signs of unnumbered "quiet pilgrimages", of uncounted people improvising odd journeys in the hope that their voyages out might become voyages in.

Perhaps, though, each era imagines itself to be increasingly on pilgrimage. As Merlin Coverley notes in The Art of Wandering, the pilgrim is among the most venerable figures of literature. The true boom-years of religious pilgrimage were, of course, medieval – but the Victorian decades saw a strong surge of interest in pilgrimage both as practice and metaphor. Hilaire Belloc's bestsellers The Path to Rome (1902) and The Old Road (1904) – the former an account of what he called his "mirific and horripilant adventure" of walking to the Holy Sepulchre – carried that interest over into the 20th century. "Pilgrimage," wrote Belloc permissively and encouragingly, "ought to be nothing but a nobler kind of travel, in which, according to our age and inclination, we tell our tales, or draw our pictures, or compose our songs."

Why the contemporary passion for pilgrimage? It clearly speaks at some level to the late-modern experience of displacement, and to the retreat of dwelling as a feasible mode of living. "Any man may be called a pilgrim," wrote Dante Gabriel Rossetti in 1874, "who leaveth the place of his birth": by that definition we are almost all pilgrims now. It represents a return of imperfectly deleted religion. And it is surely part of a broader desire to reconnect with landscape and nature, provoked by the increasing dematerialisation and disembodiment of virtualised existence. While we find it easy to say what we make of places, we are far less good at saying what places make of us – and as Rowan Williams puts it: "Place works on the pilgrim … that's what pilgrimage is for."

On pilgrimage, knowledge is not acquired unit by unit. Pilgrimage should not be imagined as a kind of epistemological orienteering, in which one's insights or discoveries are validated at punch-points along the route. No, knowledge is ideally – in Tim Ingold's fine phrase – "grown along the way": an ongoing function of passage through place, both site-specific and motion-sensitive. Because prepositions matter, we might say that while on pilgrimage people think with landscape, rather than only about it. Or, to borrow one of Belloc's absolutisms – this from a 1904 essay called "The Idea of Pilgrimage" – "The volume and depth and intensity of the world is something that only those on foot will ever experience."

Perhaps some version of this idea is why so many people seem to need the ritual walk now more than ever. In a context of the drastic privatisation of most aspects of culture, walking can offer freedoms that still escape capital's structures of credit and debt, service and obligation. The gifts offered by walking are, at their best, radical because unreciprocal. "They give me joy as I proceed," wrote John Clare simply, of field paths. Me too.

Clare was one of the best annotators of natural gift, and his poetic inheritors in this country would include Thomas A Clark, Peter Larkin, Pauline Stainer and the late RF Langley, all of whom have written uncomplacently about walking, nature, vision and – though the word remains culturally contraband – "spirituality". As such, they all stand in a tradition begun by the other great medieval work of pilgrim-literature, William Langland's Piers Plowman. Langland's poem opens with its pilgrim-narrator dressed in sheepskin, recalling his peregrinations: "I …wandered abroad in this world, listening out for its strange and wonderful events … one May morning, on Malvern Hills, out of the unknown, a marvellous thing happened to me." The idea that, despite its asperities, pilgrimage might serve as a kind of wonder-voyage, moving the pilgrim out of the verifiable and into the "marvellous', is one of its most durable attractions – and why it will surely long continue to appeal.