In 1976, two American researchers, Ellen Langer and Judith Rodin, gave a group of elderly nursing home residents a plant to care for. Another group in the same home were given plants, but told that nurses would take care of them. Three weeks later, those who had cared for their own plants reported much higher levels of happiness than those who hadn't; 18 months later, their health and levels of activity had improved and, most significantly, fewer of those residents had died.

Happiness research has come a long way since that landmark study, but what it proved has been shown time and again: having control over our lives, working towards a goal and caring for others, even for plants, can make us happier – and even extend our lives.

Researchers have since hit upon a list of activities that improve our overall wellbeing. Many of them, such as gardening, have an almost instant effect: engaging in activities in which we can lose ourselves or, in psychological terms, achieve "flow"; taking up charitable work to make us less self-absorbed and more connected to the world around us; getting physical exercise; and focusing on pleasurable pastimes that use our key skills and create greater meaning in our lives.

All these are invaluable ways to boost our mood, but achieving lasting happiness, most experts agree, depends ultimately upon understanding ourselves. If we can get to the root of who we are, and why, then true contentment will be ours. There's just the small question of how.

For Sigmund Freud and his followers, conscious thought was but the tip of the iceberg: true understanding of the self lay in the unconscious and could be unlocked only with the help of a skilled practitioner. Psychoanalysis and psychotherapy continue to rely on the notion that our wellbeing depends on continually evaluating our past.

However, there has been a significant shift in the last 10 years, born of positive psychology and the rise of the self-help industry. And it is founded on the relatively novel idea that self-awareness doesn't have to be a process of endless reflection.

The liberating notion that the solution to much of our discontent need not lie in the past forms the basis of cognitive behavioural therapy (CBT). The emphasis is on changing the way we think in the here and now, and breaking negative thought patterns. Mark Williams, professor of clinical psychology at Oxford University, uses a cognitive approach with depressive patients. One of the critical aspects that holds people back from happiness and keeps them vulnerable, he says, is their style of thinking. "They're more likely to react to a sad mood with large numbers of negative thoughts which leads to brooding and 'adhesive preoccupation' – that middle-of-the night feeling when we're caught in a cycle of worry."

If you've gone three times around the block on a problem, Williams says, then you're no stranger to adhesive preoccupation. And one of the worst preoccupations, ironically, is often, "Why aren't I as happy as I long to be?" "As soon as you start asking that question," Williams says, "it will almost certainly make you feel less happy." Instead he prescribes instant techniques to develop mindfulness, a behavioural antidote to our tendency to ruminate and catastrophise. "Mindfulness means cultivating awareness of what's going on externally and internally," he says. "It's influenced by the Buddhist idea of focusing your attention on the moment, without judgment."

Williams teaches patients to "notice" emotions and physical sensations, without reflection or analysis; to simply "see" your thoughts coming and going. "If, say, you feel tired, spend a couple of minutes asking yourself, 'Why do I feel tired? What is it about 'me' that makes me feel tired? What are the potential consequences of feeling this tired?' " After two minutes, he says, most people will feel more tired. "Now try again, but this time notice you feel tired and make space for it; allow the experience to be there, but no more."

Within eight weeks of following similar techniques, Williams says, people start to notice how the mind is drawn into fretful cycles of "What if?" scenarios, which mean we end up "living more in our head than we do in our life". In two research trials with people suffering recurring depressive episodes, mindfulness-based cognitive therapy halved the chances of depression returning.

Ultimately, mindfulness means paying much more attention to our thoughts, particularly the critical ones, and asking ourselves key questions. Would we let loved ones pass negative comment on our behaviour, intelligence or personality? Would we judge them as harshly as we do ourselves? Cognitive therapists encourage us to close the gap between our internal and external judgments, and become more accepting of ourselves.

There is one negative assumption that tends to make us more miserable than any other, Williams says; a belief that undoubtedly keeps the self-help industry afloat. "It's this tendency we all have of wanting things to be different from how they are right now. Ironically, letting go of that quest to be happy can offer a tremendous sense of relief."

American psychologist Albert Ellis, seen by many as the father of cognitive behavioural techniques, believed much of our unhappiness arises from allowing our "wants" to turn into "musts" – as in I "must" be happy. "Pretty much every time a human being gets disturbed, they're sneaking in, consciously or unconsciously, a 'must'. That's what I call 'awfulising'," he wrote. We need to become more aware of our "musts", so we can let them go.

The restless quest to alter aspects of our lives is intimately linked to self-esteem. "In evolutionary terms, we don't have to worry about lions and tigers any more; instead we're constantly vigilant about the modern-day predator, the person who will spot that we're no good," Williams says. "That generates a lot of preoccupation, and low self-esteem goes hand-in-hand."

The healthier our self-esteem, the less we tend to use words such as "me", "myself", "I" – that preoccupation with ourselves and what others think of us is an insight into how happy or not we feel. As psychotherapist Mark Tyrrell says, "Someone's mental health can be directly related to how 'self-referential' they are in their conversation – as people become healthier, they use 'I' less."

To draw us away from these negative thought cycles, positive psychologists emphasise the crucial role of focusing on the good aspects in our lives: recent research suggests that if we're grateful for what we have, we're likely to be happier, healthier and less vulnerable to depression.

As glib and contrived as it may sound, focusing on what is good about our lives is a tried and tested behavioural technique that appears to have long-term benefits. "Gratitude diaries can really work," says Dr Ilona Boniwell, a senior lecturer in applied positive psychology at the University of East London. "In studies we've found that if you manage to write down three things each day that are going well, and do it for longer than a week, it will make a difference; levels of wellbeing rise even up to six months after completing written journals."

The herculean challenge, of course, is to bear all these techniques in mind without reflecting too deeply on what we don't have, and why we are not happier in the first place – as Williams says, this can be a fast track to brooding and yet more dissatisfaction. He suggests starting with the smaller details in life: training your poorly-disciplined mind not to wander away from the present moment. "If you're drinking a cup of tea, are you really enjoying that tea or planning what you'll be doing in half an hour? The problem is, we tend to plan, and to grade life: 'When I get home from the supermarket, then I can relax'; 'When I go on holiday, that's when life is good'; 'When I'm at work, that's when life isn't interesting.' But these are all moments of your life you're not living. It turns out that if we can be present right here and now, then happiness will follow."

• What makes you happy? Join the debate