I never wanted to run an ultra marathon. I’ve always loved running, the freedom of it, the childish abandon, but I was a 10K and half-marathon runner, a “real” runner (in my eyes), someone who pushed the pace and gunned for fast times. Ultrarunning, with all its backpacks, poles and food, was not really running. It was running with all the joy bludgeoned out of it.

So when a magazine editor asked me to run the 165km Oman Desert Marathon to write an article about it, I said no: 100 miles across sand, in the heat of Oman. It sounded like hell.

But as the days went by, and I thought about it again, I began to wonder why people did these races. Ultra races are anything longer than a marathon, starting at about 30 miles, and going all the way up to the world’s longest race, a 3,100-mile jaunt around a single city block in New York.

Despite the almost incomprehensible distances, the sport of ultrarunning has been growing at an incredible rate, with around a 1,000% increase in participants globally in the past 10 years. What was going on? Why were people doing it? Was it simply to impress their friends – a symptom of our show-off, selfie-obsessed modern culture? I found that even just talking about doing one had people shaking their heads in wonder. The pictures of me slogging across the desert would certainly be a hit on Instagram. But surely there was more to it than that…

The more I thought about it, the more I began to see it not as a running race, but as an adventure. To traverse the desert on foot for five days, carrying everything I needed on my back, running at night under the stars, would surely be an amazing life experience. And who knew what I might discover about myself along the way?

So I got back to the editor and said yes. “I know it’s a bit bonkers,” he said. “Only do the first day if you want.”

But no, I now wanted the full experience. The idea had gripped me and was slowly dragging me in. I wanted to know what it was like to take on such a challenge and make it out the other side.

In the end, my experience in the Oman desert ebbed and flowed. At times I felt strong, powerful, racing like a supercharged hero across the sand. At other times I was a wreck, whimpering and on the verge of giving up. But somehow I made it to the end, to the finish on the beach, where I ran straight into the sea and, bobbing in the waves, swore I’d never do anything so stupid again.

But it wouldn’t leave me alone, this urge to push myself further, to test myself, and to come through to the other side. I started entering more and more ultra marathons. I ran 100 miles through the Pyrenees, hallucinating about cocktail parties on the side of a mountain midway through a second consecutive night without sleep. I ran for 24 hours around a running track in south London. In the end I ran 10 ultra marathons in two years. The training wasn’t too extreme. Occasionally, I would get up at 4am and head to the coast path near my home in Devon and run for four hours. But mostly the training wasn’t much more exhausting or time-consuming than before, when I was a “serious” road runner. The races themselves were when the big effort happened. My runs became longer, but slower and, in that sense, more enjoyable. I went off-road and took in the scenery. I brought snacks and sat down to eat them. The speed was no longer important.

At first, what motivated me, and kept me moving in races, was the outcome. I wanted to reach the finish line and to bask in all the relief and the warm satisfaction that came with conquering these races. The process, however, kept breaking me. In every single race, I reached a moment of crisis, where I sat down on the side of the trail and wanted to give up, where I asked myself why I was in this stupid race. The answer was always elusive and I came close to quitting many times.

I noticed that the people who did well in these races were often those with a calm, steely focus; people who seemed to be able to maintain an even emotional state throughout. I, on the other hand, more emotional than I had realised, was as up and down and as out of control as a child on a bouncy castle. I would get surges of energy where I felt I could win the entire race, skipping down mountain sides like a character in a CGI movie. And then, bam, I’d crash and tumble into dejection, anger, frustration. I would get mad at the race organisers for sending me up such steep hills. I would get mad at the hills themselves.

Seasoned ultrarunners call these moments of crisis – when you’re struggling to find the will to go on – the pain cave. And they talk about how they relish going there, how they love digging in the pain cave. But whenever I got even a few steps inside, all I could think was: “They’re all mad.” The pain cave was horrible, it ate you up, it tore you apart. Why would anyone want to experience this level of tiredness, this aching of every limb, seasoned with the thought you still had 50 miles to go?

Actually, it was often this thought that was the worst thing. The moment you looked at your watch and started to calculate how far you still had to go, the distance was usually so overwhelming that just the thought of it hit you like a punch to the stomach. Ultrarunners repeatedly told me that the key was to stay in the moment, to run the mile you’re in and forget about the rest. That sounded easy, but it wasn’t.

I read about a US runner who had lost some short-term memory function in an operation to cure her of epilepsy. As a result, when she ran, she never knew how far through a race she was, or how far she had to go. So she had no choice but to exist in the moment and just keep moving. It caused her no end of difficulties in her day-to-day life, but it made her a formidable ultrarunner. But how could I emulate her powers in my own ultrarunning? One trick was to ditch my watch, which helped a little. But I still longed for the outcome.

One runner told me his big problem was that he always thought about the finish too early. At first, I didn’t see why that was a problem. Surely that sweet feeling of conquering the race, of making it to the end, was a positive driving force. But the thought could break you, because the end was usually so far away, and because just thinking about it took you out of the present moment.

As the Japanese marathon monks of Mount Hiei once told me about their challenge to complete 1,000 marathons in 1,000 days, it was a meditation in movement. Ultrarunning, done well, was all about the process, not the outcome. And, like a meditation, that meant staying in the moment.

Eventually, I realised the secret was to embrace the struggle, as they’d told me all along. To stand in the midst of the storm, facing the oblivion, and to say to yourself: this is why I’m here. This is what I came for. A friend who ran two of my 100-mile races with me said an ultra marathon doesn’t begin until 50 miles in. “At the start,” she said, “I have too much energy, but when I get tired, everything melts away and it’s just me and the running.”

Yes, in the oblivion, deep in the pain cave, if we dig deep enough, if we push on through, we come to a place where it is strangely peaceful, where everything else melts away, even, amazingly, the pain in our legs, the tiredness that had seemed so debilitating, it all clears. And there, out in the mountains, or even on a city running track, we find ourselves fully present in the moment.

It’s rare, in our modern lives, to come this close to our stripped-back, primal selves. In the end, I found that in depths of an ultra marathon lies a stillness, an awareness of existence, that makes it all worthwhile.

It was strange, but despite initially longing for that feeling of arriving at the finish as the all-conquering hero, glowing with the satisfaction of achieving my goal, in reality the end, when it came, was often an anti-climax. I know it sounds a bit bonkers, but sometimes, despite all my struggles, I’d get to a point after 100 miles of running, where I didn’t even want to stop.

The Rise of the Ultrarunners: a Journey to the Edge of Human Endurance by Adharanand Finn is published by Guardian Faber at £14.99. Order it for £9.99 at guardianbookshop.com