Steve talks in a way that makes you want him to keep talking. His warm humour immediately puts you at ease, which makes it difficult to process what he is describing: a period in his late 20s, about two decades ago, when loneliness felt so engulfing he could barely speak. He craved the company of friends, but when they visited, he gave them cold cups of tea to make them leave. “I’d be at home absolutely desperate to see somebody, but then all I wanted was for them to get out. I’d try to get rid of them as soon as I could by not talking to them, being rude, the cold-cup-of-tea tactic – all the while knowing that was not the thing that I wanted,” he says.

This is how scientists identify lonely monkeys – they don’t look for the monkey pottering around contentedly by himself; they look for the monkey that hesitantly approaches the crowd then steps back, that makes overtures to groom another, then timidly pulls away.

We are living through an epidemic of loneliness. On Wednesday, Theresa May appointed Tracey Crouch as what some are calling the UK’s first minister for loneliness: the minister for sport and civil society will head a group that aims to tackle the problem. One recent study found that more than nine million adults in the UK are either always or often lonely. A study for the Jo Cox Commission on Loneliness found that 35% of men feel lonely at least once a week. And scientists are learning more and more about the damage chronic loneliness does to our bodies: it is as bad for you as smoking 15 cigarettes a day, as dangerous as obesity, and increases the likelihood of an early death by 26%. But what we don’t hear much about is what happens to the people who recover. Is loneliness a life sentence, or is it possible for some to break through it and come out the other side?

• “If you’ve lost confidence in being with people, I would recommend giving yourself goals (small ones at first). Talk to the shopkeeper, talk to people at bus stops. Then identify people around you who you would like to befriend. Make yourself talk to them, find a shared interest and get to the point where you can invite them to do something. You have to be really brave to invite someone for a cup of tea, but the outcome is worth the embarrassment and discomfort.”

• “It sounds a bit cliched – and for anyone suffering with mental health issues or social anxiety it can seem an impossible task – but getting out and trying new things was really key for me. Go for regular walks, exercise classes, workshops etc – anything where you might see familiar faces at the same time each week.”

• “Research all the activities that are of interest to you. Make a list of the barriers and obstacles that are preventing you from taking part, such as low self-esteem, or no one to go with. Take them one by one and think them through from every angle: what solutions are available? Can you get support from somewhere to help you do this? How can you make changes to make this possible?”

• “Think hard about whether there are other issues behind the loneliness. For me, anxiety is linked closely with my periods of isolation. Seeing a regular therapist and trying to address my anxiety head-on has helped me to prevent myself from becoming lonely again.”

• “Voluntary work of any kind will help. I volunteer for the Silver Line, a confidential helpline for older people, and I call an older lady once a week just for a chat. I hope I will be able to take advantage of that in the future myself.”

• “While face-to-face contact with people is very important, online friendships can still provide boundless emotional support, and a way to chat with people about the things you’re interested in.”

• “Get a dog! Absolutely everyone stops to talk and ask about your dog. Plus, they are great company.”

With thanks to Caroline Ryan, Jayne Matthews, Ruth and others who preferred to remain anonymous.

Steve wasn’t always lonely. His early 20s were spent socialising, dancing and DJing on the underground party scene in Leeds, with friends, music and drugs everywhere. “It was the late 80s to early 90s, the social scene was changing with dance music, it was fantastic – a whole new world opening up to me, a working-class lad from a small town,” he remembers. But his friends, he says, “weren’t dirt poor like me”; they went off to spend six months in India and came back to build careers and families, while Steve ended up in a squat, surrounded by criminality, and everything spun out of control. “I was always careful not to go too far down the drugs path, but a lot of people around me did. People were sectioned, died of overdoses; others just disappeared altogether.” One day an old friend “from the real world” came to visit and, shocked by what he saw, gave Steve £300 to get his own place. “I think he probably saved my life,” he says, quietly.

The first two weeks in that bedsit were bliss, but it did not last. He quickly grew isolated, paranoid and agoraphobic, unable even to pick up the phone to tell the landlord his toilet was broken. Antidepressants didn’t help, but after 18 months he was diagnosed with post-traumatic stress disorder resulting from his time in the squat, and a course of cognitive behavioural therapy enabled him to leave the flat for occasional temp work. He was surviving, but not living: “I did well at all the jobs, but I had no chance with the people. I was in a state close to panic whenever I had to talk to others. I felt I’d lost touch with my old friends, even the ones who were always a lifeline, because they had so much else to do. I managed to get all my worries down to just one: loneliness. I felt hopeless and resigned myself to living the rest of my life this way.”

But after three or four years of serving visitors cold tea and not turning up to friends’ weddings, something shifted. He tried to achieve one small thing every day – even just getting out of the house to buy a pint of milk. “I read lots of self-help pages on the internet with glib metaphors, likening your life to a tree, but there came a point where I just said, well, I’m not a tree, and this is absolutely no use to anybody,” he says. “I thought, soon the invitations are going to stop coming, and the only way I am going to become the social person I used to be is to actually be social.”

He made a life-changing decision: he would say yes to everything. “It was horrible to start with, especially the garden parties for their kids’ birthdays – I’d think, what am I gonna do? Look at the state of me – I’m a socially incapable freak. I was terrified. But I’d force myself to go. I was the weird guy sitting in the corner making eye contact with nobody. But I stuck with it because I knew that nothing would change without it. It was a slow and painful process, but each time it got a little bit easier,” he says. Within a couple of years, Steve felt human again.

Amy Perrin, 39, an occupational therapist and founder of the Marmalade Trust – a Bristol-based charity dedicated to tackling loneliness in vulnerable people – has watched hundreds of lonely people, from all backgrounds, take those first agonising steps. She has seen people live loneliness in its chronic form and as a transient experience; she has understood it as a character trait for some, and as circumstantial for others. She has met lonely university students, new mothers, single parents, grandparents, people with mental health problems, with learning difficulties, with disabilities, wealthy and poor – and she has also survived loneliness herself, almost a decade ago. At the age of 30, she moved to Bristol with her childhood sweetheart, but the relationship broke down.

“I suddenly found myself on my own in a different city 200 miles from my friends and family. I did the dreaded ‘lingering in the car park on a Friday after work’, knowing I wouldn’t speak to anyone again until Monday.” After a couple of months, she realised she was lonely, but couldn’t tell anyone. “I felt embarrassed – it wasn’t something that was talked about.” She tried a pottery course and joined a gym, but felt too inhibited to make friends.

It was volunteering that helped, when she started a monthly tea party for the charity Contact the Elderly. “Through volunteering and meeting other volunteers, I built my confidence,” she recalls, “and when I got to work on Monday I had something to talk about. I was able to connect with my colleagues. And shifting my focus towards other people, not being so introspective, meant my mood improved. I felt I had a purpose and a mission to help other people feel less lonely.”

It has also become her mission to reduce the stigma around loneliness by talking about it. “It has awful connotations, but it just means you don’t have the right level of social contact. If we all talked about it more, people experiencing long-term social isolation could go to their GP and say: ‘You know what? I think I’m lonely,’” she says. It sounds revolutionary.

Perrin’s experiences tally closely with what scientists at the forefront of loneliness research are discovering. Steve Cole, a professor of medicine at UCLA, studies how chronic loneliness affects our biology, and his findings suggest that having a sense of meaning in your life, being highly engaged with some kind of self-transcending goal, could help to protect against the pernicious effects of feeling isolated. He explains: “We looked at loneliness, then at sense of purpose in life, and we horse-raced them against one another and asked, if you get a person who is high in both, which one wins? Happily, it looks – at least in that particular analysis, subject to caveats – that you can be socially isolated and disconnected, but if you feel you’re on a mission, that trumps social poverty.”

This is crucial in thinking about how individuals can work through loneliness. For Cole, “Lonely people don’t choose to be lonely; they’re often lonely as a defensive measure against a world that they perceive to be threatening and hostile. You really need to change this worldview.” His study suggests that one way to change that worldview is by trying to change the world.

There is no doubt that Perrin’s work has helped to lift her out of loneliness. “I remember one man, he was in his 90s, and his voice was really, really croaky because he hadn’t spoken to anyone for more than three months. He became incredibly tearful talking about how excited he was that he was going to have company. I still find that very emotional,” she says. He is not alone in his loneliness: according to a survey by Age UK, 360,000 people aged 65 and over have not had a conversation with friends or family for a week, and 200,000 have gone without for a month.

When Doreen Fairclough, 79, broke her shoulder, she was told she had to go into a nursing home for six weeks to recover, but she ended up staying for almost nine months. She couldn’t go back home because she could not have managed on her own; she had never married and, having nursed her mother and then her brother, who both died of cancer, now had no one. Fairclough just wanted to have a chat and go to Morrisons, but the other nursing home residents were often asleep, and she wasn’t allowed out alone. Then she heard about Homeshare, a charity that, for a monthly fee, matches people who need companionship at home with others who are seeking affordable accommodation and who agree to provide about 10 hours of support a week, as well as overnight security.

Last year, Fairclough moved back into her own home in Lancashire, which she now shares with Lucille, 23. They go on shopping trips and to the cinema, and they went to the pantomime together at Christmas. “It has changed my life,” she says. “I have company, and I’m at home. It’s not nice being on your own – you’ve nobody to talk to. I feel all right now.”

But loneliness is not always and not only a question of social isolation, and the way out is not necessarily through other people. Sometimes you have to look inwards. That is what Diana Villegas, 25, found when she realised she felt lonely in her relationship. She met her boyfriend while studying abroad in France – they were conversation partners; she practised her French with him, while he practised his English with her – but it wasn’t until Villegas moved to Germany for work that their relationship began, four years ago.

She initially attributed the sense of loneliness she had to the long-distance nature of their relationship, but it stayed when he moved to Germany to live with her. She felt distant, unable to connect. “Someone can be there next to you, but you don’t feel you’re actually with them, you don’t feel any connection. You feel lonely, and guilty because you feel lonely, and very unsure about where you stand. I felt an immense insecurity in myself, in my relationship, in what I should be doing with my life,” she says.

At first, she panicked. “I was fed this idea that everything’s going to be fine once you find someone who understands you, and you’ll never be lonely again. But that is such an unrealistic expectation. In my past partnerships, I went along with this idea; if I hadn’t made a change in my thinking in this relationship, I would have continued making this mistake over and over again.

“To be completely honest, at the very start, I thought I was in the right and he was in the wrong.” Then they talked, and started to see each other’s point of view. Villegas realised they had completely different expectations: she was used to communicating with family, friends and past partners frequently throughout the day, while he was not. “There are no right or wrong communication expectations, so the first step for us was to find common ground. How often did I expect to talk? How would this take place? Who would initiate it? What came afterwards was a lot of trying, adjusting and learning how to be respectful towards different points of view.” He learned to share his problems with her instead of bottling them up; she learned to give him space; they agreed to check in with each other at the end of every day to talk about how they were feeling.

They now live together in France, and although she still feels lonely from time to time, she now knows what to do about it. She has also started going to the gym regularly to let off steam, and has joined expat social groups to make friends in a similar situation. “It’s important to find other ways to deal with loneliness and accept that just because you feel like that, it doesn’t mean your partner is at fault. It’s normal to feel lonely sometimes,” she says.

None of the people I speak with say that after enduring their period of extreme loneliness, they never felt lonely again; but now they know it is transient – not pleasant by any means, but bearable. For Steve, it is a little like coming back from the dead. “Today my life is entirely different,” he says. “I’ve got a good job with people I love, who respect me and see me as a source of endless stories from a life that few have lived and fewer have survived.” Though he would like to have a partner, he has made peace with the fact that it might not happen. “I’ve reconnected with all my old friends, and their kids see me as their naughty uncle, even the ones who are grown up now. It’s a great feeling and it goes some way to making up for the fact I’ll probably never have a family of my own. I’ve come to accept the idea of living on my own – alone, rather than lonely.

“Whatever social foibles I’ve still got, they’ve been folded into all the friendships I have. Everyone I know is a part of my family, and I love them all for it. Loneliness is a beast that will drown you if you give it the chance. If you have to change to overcome it, don’t fight the change. Life is change, after all.”

In the UK, Samaritans can be contacted on 116 123.