A few months ago, I spent an evening sitting on the sofa in my flat, cropping my head out of a series of wedding photographs. It was a fairly surreal experience excising my smiling face from the pictures taken outside the chapel. It was not something I had ever anticipated, because you don’t think about divorce when you’re walking down the aisle. You don’t imagine it will happen to you. You don’t believe that one day, you will be digitally altering your wedding photographs so that you can sell your mermaid-style gown and long-sleeved lace bolero to a stranger on eBay.

And yet this is where I found myself. The dress had been hanging in my wardrobe for three years since the end of my marriage. It had been pressed up against the winter coats, shrouded in its dry-clean carrier, and although I tried to forget about it, I never could. The dress took up residence like an unwanted tenant, a constant reminder of my failure.

As I posted details of the dress online, I began to think about failure and its shadow-twin, success. If I listed the achievements of my life, on paper, without emotion, I would have to concede that on some level I am doing OK. I have published four novels. I make a decent living as a journalist. I have a good degree, a wide friendship group and I pay my rent promptly each month.

I am, I suppose, objectively successful. But it doesn’t always feel that way. I still hadn’t grown out of the habit of walking into bookshops and rearranging my novels so they were on the top of the display table piles. Surely no truly successful author does that.

The more I thought about it, the more I realised that the biggest, most transformative moments of my life came through crisis or failure. They came when I least expected them, when I felt ill-equipped to deal with the fallout. And yet each time, I had survived.

Into this category I put the fact that I had got married at the age of 33, but divorced three years later. I had tried to have children, but failed despite two rounds of IVF and a natural pregnancy that ended in miscarriage at three months. I had been in and out of relationships that never seemed to last.

Professionally, too, there had been some knocks. I had written a heartfelt second novel about war and its impact, loosely based on the death of a former boyfriend in Iraq. It had been an important book to me, but when it was published it barely seemed to register. The few people who read it were kind and I got used to deflecting my wider sense of rejection with humour: “It’s a beach read,” I told tiny literary festival audiences, “if the beach is Dunkirk.”

I called that book Home Fires. When Kamila Shamsie won the Women’s prize this year for her critically acclaimed novel, Home Fire, it was hilariously ironic: in my head, the success of her work highlighted the failure of my own. But the truth was, I grew from the experience. Afterwards, I put what I’d learned about writing into two more books. The latest, The Party, became a bestseller. Was it, I wondered, that I had unwittingly become successful as a by-product of failure? Had I, to paraphrase the words of the late American playwright Edward Albee, succeeded interestingly precisely because I’d failed interestingly first?

It was a topic that interested me enough to start asking other people the same questions. In my day job as a journalist, I’m in the privileged position of interviewing a lot of celebrities whose fame is a distinctive marker of a certain type of success. Yet they all had their own tales about life going awry. The actress Natalie Dormer, 36, recently told me that her 20s had been a decade of “self-doubt and anxiety”. She said she was relieved to reach her 30s because “You’ve fucked up. When you’ve fucked up a number of times, hopefully the idea is you don’t fuck up as badly the next time when you’re presented with the same or a similar situation. I’m strong because I’ve been weak, I’m wise because I’ve been stupid.”

This seemed to me to be at the root of it: how one becomes strong because of weakness; how one is more likely to succeed if one has learned from failure. In recent years, the notion of “failing well” has gained considerable currency. Books such as Adapt: Why Success Always Starts with Failure by Tim Harford and The Art of Failing by Anthony McGowan (which describes itself as “a chronicle of one man’s daily failures and disappointments”) have added grist to the notion that failure can be distilled into something more positive if the right alchemy is applied. Harford argues that improvising rather than planning is the way to tackle everything from terrorism, climate change, poverty and innovation to the financial crisis, and that trial and error is the best way of achieving long-term solutions to complex problems. “No plan survives first contact with the enemy,” he writes. “What matters is how quickly the leader is able to adapt.”

It was an idea worth exploring. So I began a podcast series called How to Fail. The idea was for me to do eight one-to-one interviews over eight weeks with highly successful people about what failure had taught them.

The rise of social media means we now live in an age of positive curation, where Instagram feeds and Pinterest mood boards are designed to give the most glowing impression of our lives. In this context, failure doesn’t get much airplay. But, I thought, wouldn’t it be refreshing if we stripped back the carefully crafted layers of our supposedly perfect selves, and revealed ourselves to be vulnerable?

To my delight, people seemed to warm to the concept. Soon, I had a wonderful roster of participants, plucked from friends and contacts, including the actor and writer Phoebe Waller-Bridge, the novelist Sebastian Faulks, the political activist Gina Miller and the writers Olivia Laing, David Nicholls, Dolly Alderton and Sathnam Sanghera.

Before each recording, I asked my interviewees to come up with three examples in their life when they felt they had failed and that they were willing to talk about. The failures cited ranged from humorous accounts of flunked exams to the big, life-altering crises of divorce and serious illness.

It was fascinating to see how men and women had different attitudes. Many of the men I approached balked at the idea they had failed at anything. They cited lost tennis matches, unrisen soufflés and the inability to play a musical instrument. The women routinely responded that they would have trouble whittling down their myriad failures to just three instances.

“There are so many to choose from!” said Olivia Laing. “Women are so socialised to be self-deprecating, to not claim their successes… I think it’s much easier for women to say, ‘I didn’t do well at this,’ than to say, ‘I did marvellously at it.’ Which is a bit depressing really, isn’t it?”

It is. But I was touched by how openly the women I interviewed were willing to delve into their failures. Gina Miller, the woman who took on the government over Brexit and won, wanted to talk not about that victory but about surviving an abusive marriage, raising a daughter with special needs and failing to graduate from a law degree.

Coping with these twists in life, said Miller, “taught me that you mustn’t make such a rigid plan for your life that when it doesn’t work out, you’re so sad and you then just live feeling this sense of disappointment, because that ruins the rest of your life. You have to let go. It’s like a mourning. You have to grieve and put it aside and bury it and then move on, and that’s what I learned to do.”

The consequence of all of this was that she was forced to be honest about her own mistakes and weaknesses and, by confronting them, she built up the emotional resilience necessary to tackle the next challenge. Success, she explained, was not about getting things right the first time, but stemmed from being able to look at one’s past honestly and then to correct missteps or errors of judgment. That, in turn, gave her more confidence to make brave choices. “In life we’re all going to fail,” she said. “So you might as well have a strategy for how you deal with failure, and then once you’ve got that in your back pocket, you can go out in life and really take risks.”

Laing admitted that she saw the whole of her 20s as a failure – she struggled to find her way, trying out several versions of herself, as a road protester, a medical herbalist, a cleaner and the deputy literary editor of this newspaper before she left and stepped into an unknown future. Not knowing what else to do, she funnelled her distress into the proposal for a book about Virginia Woolf, weaving in elements from her own life.

That book became To the River, which was shortlisted for the Ondaatje prize, and Laing has since gone on to become one of our foremost writers and cultural commentators, who often uses her own experiences of alienation and loneliness as a means to get to some essential, connecting truth. Her failures have, in this way, become her art.

Sebastian Faulks, the author of 16 books including the international bestseller Birdsong, at first felt he wouldn’t have much to contribute to the podcast. His attitude towards failure was that it was a matter of how one perceived it, and he felt his life thus far had been blessed rather than cursed. He gave me three, deliberately playful examples of failure including “once getting out [at cricket] when I had made 98 and chipped a return catch to the bowler” and the occasion on which he came second in a prestigious Italian literary prize. It was awarded, instead, to the brother-in-law of the chairman of the judges.

“Is that a failure?” he mused. “I mean, I wouldn’t have thought so, I thought it was rather a success to be going to Milan to be celebrated in a country not your own for a book with no Italian connection.”

But Faulks had also experienced periods of depression in his life, most notably at university where he “struggled to adapt… I was extremely confused and very fragile and it took quite a lot of time to get over that. I wouldn’t say I have got over it really.”

Many of my interviewees’ failures stemmed from doomed romantic relationships, but they often credited those experiences with providing stimulation for creativity. Phoebe Waller-Bridge, who won a Bafta for writing and starring in the feminist sitcom Fleabag, found a certain “glory in failure [because] fighting so hard to be so in love with someone with all that passion in your 20s and teens and then throwing everything at it and it’s not working, or there being so much pain – that is the stuff that so much creativity comes out of. So it’s out of those painful break-ups or miscommunications – or just horrible sticky one-night stands – that you grow in those moments, and so I value them all.”

With so much of my time being spent asking other people about their failures, it was only natural that my thoughts would wander to my own.

If I had to list my three major failures, right up at the top would be the failure of my marriage. Part of the reason my marriage ended (and this will only ever be a subjective assessment) was that I think I tried too hard to please. I forgot, in the rush to appear flawless and irreproachable, that it was far more important to be real than perfect. Like many women I know, I spent my 20s desperately wanting to be loved in order to shore up a shaky sense of self. If I placed no demands on my spouse, the internal reasoning went, if I did everything right, then there would be no excuse not to love me.

It’s terrible logic and, inevitably, it fell apart. The divorce catapulted me into a different sort of life from the one I had imagined. Here I was, in my late 30s, single, without children, and navigating uncharted waters. Despite never having thought of myself as a particularly unconventional person, it struck me I was living an unconventional life. My failure to have children at the time when all my contemporaries were having babies and moving closer to good schools made me reassess what I could get from the life I already had. If motherhood wasn’t going to be part of the future I had always imagined for myself, where else would I find fulfilment?

Life crises have a way of doing that: they strip you of your old certainties and throw you into chaos. The only way to survive is to surrender to the process. When you emerge, blinking into the light, you have to rebuild what you thought you knew about yourself.

It dawned on me that I had my work. I was lucky in the sense that being a writer means you never feel fully alone – you always have the company of the characters you create. I also had my friends and family, from whom I get a great deal of love and compassion. And, actually, if I looked at the failure in a different way, it could also double up as an opportunity: I was free of responsibility. I was no longer living my life in a misguided attempt to please other people. So I could live in a more agile, flexible way. If I wanted to move to Los Angeles for three months and live in an Airbnb, then I could (and did).

I think what I’ve learned from failure is that things not turning out the way you’d planned gives your time on this earth a lot more texture and meaning. I’m now oddly grateful for all the losses – the miscarriage, the divorce, the subsequent break-up of another relationship – because without them I wouldn’t be who I am or where I am, and I wouldn’t have seen the richness in a different kind of life.

Besides, no failure is all-consuming. A nice woman in Shropshire bought my wedding dress on eBay and I put the money into funding the first couple of episodes of the podcast. As I folded the dress into a box, wrapping it carefully in layers of tissue paper, I thought to myself that this wasn’t a failure at all. It was a part of my life. I had learned from it. And now I was letting it go.

How to Fail with Elizabeth Day is out on iTunes now

• This article was amended on 15 July 2018. An earlier version said Olivia Laing was a homeopath. This has been corrected