Last year, my New Year resolution was to go for a run first thing every morning. It started well: 1 January was a great success. On 2 January, though, I hit snooze and went back to sleep. I tried to get it going again, I really did – I even wore my gym clothes to bed – but nothing worked.

This year, I’ve resolved to wean myself off scrolling mindlessly through social media on my phone, but when it comes to making resolutions – or, rather, breaking them – it feels as though there are forces at work far stronger than my willpower. I know I’m not alone in that; if I were, there wouldn’t be nearly 6,000 books on Amazon under the category “self help – habits”, nor so many psychologists researching the subject. So, could they help me keep my resolution this year?

Charles Duhigg, the author of The Power of Habit, certainly thinks so. He tells me there is “a ton of research” to show that New Year resolutions are an effective way to make changes: they create a sense of expectation and ceremony, while the link to a particular day helps to fit our experiences into a narrative of before and after, which makes change more likely. “There are people who will decide on 1 January to lose two stone and who will keep it off for the rest of their lives, others who have been smoking two packs a day for over a decade who will decide to quit and who will still not smoke this time next year,” he says. “Anyone can change any habit; it doesn’t matter how old you are or how deeply ingrained that behaviour is. But that doesn’t mean – as everyone knows – that New Year resolutions are consistently successful.

When our New Year resolutions fail, we berate ourselves for our weak self-discipline; we tell ourselves our willpower wasn’t strong enough, as though we are a marathon runner who couldn’t make it to the finish line. The image of a self-control muscle that gets tired over time, first proposed by the social psychologist Roy Baumeister in the late 90s, has shaped our collective consciousness.

But a new generation of psychologists, unable to replicate the studies that proved his theory of “ego depletion”, are questioning this model. They are exploring other factors that might determine whether individuals can stick to their goals, including their motivation and environment, explains Katharina Bernecker, a postdoctoral researcher at Leibniz-Institut in Tübingen, south-west Germany. “The idea of a limited resource is about a capacity or an ability – you can or you can’t – whereas motivation is something that fluctuates. We’re searching for a new theory that tells us more about this process,” she says.

Putting all our New Year resolution eggs in a willpower basket is exactly where we are going wrong, suggests Wendy Wood, provost professor of psychology and business at the University of Southern California. Although studies show that people who have a lot of self-control tend to be good at meeting their goals – if they are motivated to do well at work, they get promoted; if they want to live a healthy lifestyle, they exercise more – it isn’t because they use their willpower to control their behaviour, she explains. In fact, it is because they find a way around it. These individuals score highly on scales that measure their ability to control their actions and resist temptation, but “the interesting thing is that it doesn’t work that way”, Wood says. “What we’ve learned is that people with high self-control are not going through these white-knuckle struggles to eat better, exercise more or work harder. Instead, what they do is form habits. They automate their behaviours that get them to their goals, so they perform them without even thinking about it. That’s what makes them so successful.” It isn’t about willpower; it is about habits.

This epiphany is what turned Gretchen Rubin, the author of the bestselling blockbuster Better Than Before, into the US’s happiness queen. “Habits are freeing and energising and really powerful. If there’s something you want to do consistently in your life – like New Year resolutions – habits can make the wear and tear on following through so much easier. They get us out of the tiresome business of making decisions and using our self-control.”

Rubin speaks with the authority of a woman who has honed her lifestyle by knowing her weaknesses and how to overcome them: “I don’t have to decide to get up at 6am – that’s a habit for me, on autopilot,” she says.

So, what makes a habit? First, says Bas Verplanken, professor of psychology at the University of Bath, they are automatic, occurring as part of our daily flow. “If going to the gym is a conscious decision, we’re vulnerable, because we have a fantastic capacity to rationalise why we should not go – we’re very, very good at that. Habits protect you against thinking,” he says. Second, they are triggered by cues in the environment, such as time or place. Third, every habit has a reward: when our brain starts to anticipate and crave the reward, it makes the behaviour automatic.

Thanks to the boom in research in the field in the past decade, Duhigg tells me, “we’ve seen a golden age in understanding the neurology and psychology of habit formation”. In the first half of the 19th century, psychology research focused solely on observable behaviour – as opposed to what Verplanken calls “what is going on under the hood” – a period known as the behaviourist revolution, led by the psychologist BF Skinner. This was followed by the cognitive revolution, which investigated how we think, as opposed to how habits work, which entails investigating how we can avoid thinking. “It’s only since the turn of this century that we’ve started to realise that the brain is actually made up of multiple systems that are connected, but somewhat separate as well,” says Wood. “One of these is a neural system that learns in a habit way and this is represented in our behaviour in terms of automaticity. All of a sudden, habits started to gain credibility.”

It was neuroscientists who brought habits on to psychology’s radar, since brain scans cast light on mechanisms unfolding in the deepest, darkest recesses of the brain, identifying which parts are activated as a behaviour becomes habitual. “As we repeat actions, we engage different aspects of our neural system and you can actually see habit formation taking place in the brain,” says Wood. “When you have people in scanners, activation starts in the decision-making areas of the brain – the prefrontal cortex and the hippocampus. Over time, as you repeat a behaviour and keep getting that reward, activation shifts more to the basal ganglial areas, particularly the putanem, because we’re no longer thinking actively; instead, we’re responding based on habit.” Wood’s research shows that 43% of what we do every day is performed out of habit. “It’s a shortcut – if you do what you did before, in this context, you’ll get the reward that you got before,” she says.

These insights have significant implications for our New Year resolutions, says Duhigg. “Every habit has three components: the cue, the routine itself and the reward. A huge part of understanding how to change or control your habits is diagnosing the cues and, most importantly, the reward that routine delivers to you,” he says. I cast him in the role of my personal resolution consultant and ask what I need to do to break my phone habit. “The first thing is the terminology,” he says. “Breaking a habit is almost impossible. Once the neural pathways are set with cue, routine and reward, they are there to stay.” Rather than thinking in terms of breaking a bad habit, he says, I need to change my habit by finding a new routine that corresponds to the old cue, one that will deliver whatever reward I am getting from it currently.

I figure out that my cue is flopping on to the sofa after a long day, but I can’t pinpoint what reward it gives me. I suppose there is a voyeuristic pleasure in looking at my friends’ photos on Facebook and I’m interested by articles linked to on Twitter. This is a good start, says Duhigg. “You can easily change this habit; you just need to spend some time experimenting with other routines to see what can deliver something similar to that old reward,” he says.

For Rubin, the answer is more complex. “There is no magic, one-size-fits-all solution. Otherwise, we would have figured it out,” she says. In her book The Four Tendencies, she divides people into categories based on how we respond to inner and outer expectations (there is a quiz on her website, if you are intrigued). She says we need to tap into our own tendency to learn how to best stick to our habits. She diagnoses me as an Obliger – someone who meets outer expectations, but resists inner expectations; sounds about right – and suggests that the reason I failed so miserably (not her words) at keeping my running resolution last year could be that I had no outer accountability. Had I arranged to meet a friend for runs, turning inner expectations into outer, I might have been more successful. She also questions why I decided to run early in the morning when I usually exercise in the evening and why I chose running when I prefer exercise classes. Now, I realise why it went wrong: I didn’t know myself.

I wonder if this is the root of most failed New Year resolutions. The difficulty making or changing habits then becomes a more profound question of why we can’t make ourselves do the things we want or stop ourselves from doing the things we don’t. According to David Bell, a psychoanalyst and consultant psychiatrist at the Tavistock and Portman NHS Foundation Trust, the answer is resistance. “One thing Freud put on the map is that we’re all much more resistant to change than we like to believe,” he says. Resistance is what happens when our unconscious holds us back from making the changes we consciously desire. Resolving to behave differently won’t help, he says; what can help is trying to figure out why we are the way we are.

“Sometimes people are very fearful of others for reasons they don’t understand,” he says. “That tendency in their character is not going to be dealt with by deciding not to be frightened of other people – they need to understand the roots of that and that takes more time.”

This is why thinking of personal growth as a neurological habit loop to be hacked doesn’t sit well with Bell. “It might be helpful for some people, but I think human beings have a rich subjective life, some of it conscious, a lot more less conscious,” he says. “To me, this sounds like treating your own mind as if it’s a machine that needs correction. I think that mechanical way of thinking about the self is alienating to individuals.” He sees a darker side to this kind of self-improvement strategy, describing a cultural shift that has intensified over the past 20 years, in which there is an increasing pressure, he says, “to think of individuals just as individuals, not as people in a network of social, cultural, historical relationships that are determining and affecting them. That goes hand in hand with a wish to transfer all responsibility to change on to the individuals themselves – and with undermining systems of social and medical welfare that form the basis of our way of thinking about our responsibilities for each other.”

Am I any more likely, I ask Verplanken, to be able to stick to my New Year resolution today than I would have been 100 years ago, given all these advances in psychology? His answer surprises me. “Human nature is, in essence, not very different from 100 years ago – we are wired in the same way,” he says. “We like to think that we are progressing – and we are in all kinds of ways – but human nature doesn’t change.”

Yet I have felt something shift after speaking to these experts. Why do I reach for my phone every evening? Because I need to escape my stressful day and it is an easy way out – it brings relief. So, I am following Duhigg’s advice and experimenting with different behaviours that could give me the same reward – watching TV, reading a novel, listening to a podcast – and I am also, as Bell suggests, trying to explore more deeply why I get so anxious in the first place.

I hope that, eventually, this will enable me to connect better with myself and with my loved ones, instead of choosing to disconnect from the world. It is ambitious, as New Year resolutions go, but I am feeling optimistic.