Seventy-three years after it came out, and 54 years after its author died, How to Win Friends and Influence People , a motivational guide written by an unemployed salesman-turned-actor called Dale Carnegie, is back in the bestseller charts: astonishingly, and for reasons that are not immediately clear, it is, this week, the eighth bestselling self-help book in Britain.

There's no movie tie-in, it's not even a new edition, and although other books written in 1936, are still read today, people don't tend to turn to Young Men in Spats by PG Wodehouse for tips on how to negotiate contemporary society, or Gone With The Wind for guidance on race relations. And yet, a book that opens with details of a recent news story (a 1931 police stand-off with a robber called "Two Gun Crowley"), written by a man born in rural Missouri when Queen Victoria was still on the throne, is still regarded as containing the sort of universal truths that, even in these difficult times, is worth a cover price of £8.99.

But then, of course, it's the difficult times that provide the link. Carnegie was writing during the Great Depression, as was his immediate successor Napoleon Hill, whose 1937 book, Think and Grow Rich! , is considered another early classic of the self-help genre. These were days in which men needed to learn to help themselves because there was no other type of help available.

And here we are once again. The recession has ripped through publishing as it has all aspects of British life. Philip Stone, an editor at The Bookseller, says that book sales are down by one per cent so far this year - "which, all things considered, is not too bad" - but that mind-body-spirit, of which self-help makes up by far the greatest part, is up by 25%. This week's top 10 contains not just Dale Carnegie's work but Susan Jeffers's Feel The Fear and Do It Anyway , first published 20 years ago, and the decade-old Who Moved My Cheese? by Spencer Johnson, a book that companies tend to give employees when they're "letting them go". Fiction dates, non-fiction dates, but self-help books are more like sacred religious texts, discovered afresh by each new generation.

And it's not just the books that are booming. "Salons" are cropping up all over the country. In Bloomsbury there's the new "School of Life" - set up by a group including Alain de Botton to "offer good ideas for everyday living". And when Malcolm Gladwell, the highly regarded New Yorker writer, turns up in Britain to lecture on success, hundreds come to listen in respectful silence. We live in self-improving times.

Michael Neill, a "success coach" who has worked for years with Paul McKenna, far and away the number one bestselling self-help author in Britain today, says that, 10 years ago, "self-help books just weren't considered quite 'normal'. You couldn't read them in public. It wasn't quite that you had to hide them inside a copy of a porn mag, but almost."

Now it's not just "normal", it's mainstream. Positive thinking is everywhere. Where you expect it, where you don't expect it. Last Wednesday Harriet Harman told the Labour conference to forget the Sun 's disavowal of the party. "Don't get bitter," she said. "Get better." It's a self-help staple, but then this is a language which has entered the corporate and business world, and the corporate and business world has returned the favour. It's become the dominant discourse of much of public life, so ordinary and everyday we barely even notice it.

Positive psychology, the so-called new "science of happiness", has, in just 10 years, become a cultural orthodoxy and a burgeoning field of academic study. It's the single most popular course for undergraduates at Harvard, and in Britain it has been instrumental in persuading the government to back large-scale funding of CBT (cognitive behavioural therapy). The application of science has given self- help a rigour and respect that for years it could only dream about.

But on the other hand there is still what Maureen Rice, the editor of the glossy mag Psychologies , calls "an awful lot of crap out there". There is. It's true. Not just a rash of recent books about angels ( Angels in My Hair , Angels in Your Pocket, How to Hear Your Angels ), which are the new misery memoirs flying off the supermarket shelves, but also a bizarre and insanely popular concept called the "law of attraction" in which, basically, if you ask the universe for a new car or a new boyfriend, the universe will provide.

Psychologies was launched four years ago and competes on the news-stands against Marie Claire and Elle , and Rice is an enthusiastic proponent of positive psychology. But the main part of her job is, she says, "reading rubbish so you don't have to". But then it's the broadest of churches, self-help, as I find when I turn up for a two-and-a-half day event in central London called "I Can Do It!", organised by Hay House, a specialist publisher in the field. Last weekend's session was the first of what is to be an annual event in the UK (America, Canada and Australia already have their own versions).

I assume I've got the wrong queue when I arrive. It seems more like the kind of crowd you'd find outside a designer sale: diverse, with a scattering of men, but a preponderance of glamour blondes in expensive heels and haircuts. Inside the lecture hall - it seats 900 people but it's been sold out for months, and Jessica Crockett, the director of marketing, tells me "we could have filled it at least three or four times over" - there's a low hum of expectation. At £299 for the weekend, not counting hotels or travel, it's a significant sum to invest in simply hearing a few people speak.

Except it's more than that. Or, at least, the audience are hoping it will be. Among the first people I talk to are South African sisters Candice and Karyn Velleman, 27 and 36, who both work in investment banking. "This is our outlet," says Candice. "This is how we survive in the corporate world. It's just recognising that everything is in your head and that your thoughts create everything in your life."

Which is more or less what Napoleon Hill had to say back in 1937. "What the mind of man can conceive and believe," he wrote. "It can achieve." And, in his case, this is how it seemed. He was a presidential adviser to FD Roosevelt from 1933-6, and it's not impossible to trace a path between Hill's beliefs and the high-minded ideals that underpinned the New Deal: the politics of optimism and self-reliance.

A few minutes later, Louise Hay, the founder of Hay House, bounces up on stage and says almost exactly the same thing (although she gets the biggest cheer when she announces, simply, "I'm 83 years old!"). And, immediately after her, it's what Wayne Dyer says too, although he takes three hours, and throws in lots of jokes and stories about his childhood in an orphanage, and his six daughters and two sons, and how this summer he married the chat show host Ellen DeGeneres to her actress girlfriend, Portia de Rossi. Finally we learn that he has recently been diagnosed with leukemia. A sympathetic murmur ripples through the crowd.

"Don't go 'Ah!'" he says. "It's not sad. I'm not sad. All you have to do is change your imagination of yourself. You can assume yourself to be wealthy. You can assume yourself to be healed. You can assume everything in your imagination. You have to Assume The Feeling of The Wish Fulfilled. If you are writing anything down tonight, write that down." So we do. I'm not the only one with a notepad - all around me women are jotting down key words and phrases in pretty floral notebooks.

Louise Hay wrote and self-published her first book, You Can Heal Your Life , 25 years ago, and it's sold a staggering 35m copies around the world since. And Dyer is another celebrated name in the field: he wrote his first bestseller Your Erroneous Zones in the 1970s and has gone on to publish more than 30 others. This isn't just big business, it's huge business. But then self-help and capitalism are the most intimate of bedfellows.

Tim Harford, the Financial Times 's "Undercover Economist", says there's a huge amount of speculation about how people's behaviour changes during a recession but that certain things are known, one of which is that we start to believe that success in life is more dependent on luck. "Which is where the angels and the cosmic ordering stuff comes in." It makes people wonder about how much is down to them "and how much is down to the universe".

Over the course of the weekend I meet life coaches, therapists, health professionals and charity workers, and a whole rich seam of ex-City professionals like Jacqui Cowing, 42, from Northants. She was earning "a six-figure salary" as a director in a corporate recruitment company. "And then in January I just gave my notice in. I felt there was a lack of integrity in the corporate world. I was being asked to do things that just didn't fit my moral code."

The fall-out from the financial crisis and its ongoing uncertainties has rippled out in all directions, raising questions about what it is to be successful. To be happy. Even what it is to be human. When I speak to Alain de Botton, whose books address similar questions, albeit in a very different way, he says: "I think what the financial crisis has done is to show that there's a vacuum of ideas at the top of society about how society should be run. These people were enormously confident about how things should be. Gordon Brown and Alan Greenspan were lecturing us endlessly about it, and basically they've been shown to have very little clue. And this has opened up a huge amount of imaginative space. People are actively looking for new ideas."

In the past, people turned to philosophy. "The interesting thing is that if you read Seneca or Cicero or Epicurus, what they're doing is not a million miles away from the modern self-help book. They're trying to fill practical goals in life, to help you survive worries of death and the vicissitudes of life."

But what de Botton calls "elite culture" has, he says, "abandoned a project on which it was engaged for most of human history. English literature, philosophy, history, they used to understand their role as basically being about the nourishment of the soul. But they've abandoned that field, leaving the area open to what are largely second-rate minds."

It's almost impossible not to be seduced by the positivity that is all around, all weekend. It's not just the speakers, who range from Doreen Virtue, a leading light in angel circles (she speaks to the archangel Michael, who tells Bridget in the audience that she has a wheat allergy; he's a very specific sort of archangel), to Robert Holden, a respected psychologist who used to work in the NHS.

Holden is typical of the new breed of positive psychologist. "I studied psychology for six years," he tells us. "And we studied paranoia, suicide, depression, eating disorders, psychoses, neuroses. The thing we didn't study was happiness. We had only a one-hour lecture in six years."

Happiness, he says, is what helps you to lead "an authentic life", it's what can help you live "in the now" rather than deferring to a future that never comes. And to prove the utility of joy he plays the Stavros Flatley clip from last year's Britain's Got Talent . "Sometimes in order to be happy in the present moment," he says, "you have to give up all hope of a better past."

In fact, almost all the speakers say this. It's one of the key messages of both self-help and psychotherapy, and in this case it's allied to other techniques: counting your blessings and looking on the bright side being two of them. And for the audience, it seems to work, at least momentarily. They leave Robert Holden's lecture looking, well, happy.

But it doesn't necessarily work for all people in all circumstances. When Barbara Ehrenreich, the left-wing American journalist and essayist, was diagnosed with breast cancer she found herself entering a world not just of painful medical procedures and agonising decisions but also of what she calls "bright-siding". She was exhorted to think positively, an orthodoxy so powerfully unremitting that "it remains almost axiomatic in breast cancer culture that survival hinges on 'attitude'".

It's not just that those diagnosed with cancer are encouraged to maintain an upbeat attitude towards treatment - Ehrenreich found she was asked to believe that her cancer was, in some way, good for her. That it would deepen her personal relationships and transform her attitude to life. Nancy Brinker, the founder of one of the US's largest breast cancer foundations, was typical, she found, in claiming that she "had come out stronger. With a new sense of priorities". Cancer had made her "happier than I had ever been in my life".

It didn't make Ehrenreich happy. It made her angry. And the result is a new book, Bright-sided: How The Relentless Promotion of Positive Thinking has Undermined America , published next month in the US and in January in the UK by Granta as Smile or Die . In it, she traces the history of "optimism bias" and its outcomes, not just on individuals but on nations. The refusal to recognise the warning signs before 9/11. The "reckless optimism" that characterised the invasion of Iraq. And, most damningly of all, the wild, free-spending, it-can-never-happen-here corporatism that directly led to the global financial crisis.

In an economy overseen by optimists, house prices would always go up, stock markets would never crash. Positive thinking became not just the language of the mainstream but, on both sides of the Atlantic, political dogma and economic principle too. An ideology that originated in America has fanned out across the English-speaking world, and from there to everywhere else, hand-in-hand with the doctrine of free-market economics. Wayne Dyer's 30-year-old bestseller, Y our Erroneous Zones , is currently No 1 in South Korea ("and we have no idea why," Reid Tracy, Hay House's CEO tells me). Louise Hay is a household name in Russia.

It's globalisation by any other name, according to Eric Wilson, a professor of English, who wrote a book called Against Happiness . "The self-help movement has attempted to commodify experience," he tells me. "It's intimately tied into capitalism. Buy this package and, almost like a technology, it will move you forward with the goal of a trouble-free life."

In fact, critics of self-help are as thick on the ground as self-help authors are, sometimes managing to be both simultaneously. When I call Oliver James, the psychologist and broadcaster, he's damning about the genre, despite having written various books that sit within the Mind-Body-Spirit section including "They F*** You Up" and "Affluenza".

"It's snake oil," he says, "and I explicitly reject it. Positive psychology and cognitive behavioural therapy and the idea that anybody can be anyone are American ideas involving what's basically a sort of magical thinking. The purest example is The Secret, which is a disgraceful book. It's just wicked really. It doesn't have any kind of basis whatsoever. It says: if you want something you just have to wish for it, like my four-year-old does. It's a kind of psychology for toddlers."

The Secret , written by an Australian television producer, Rhonda Byrne, is a leading proponent of the "law of attraction". It was endorsed by Oprah, became a massive global bestseller and has spawned a host of imitators. I go to a talk by a German author called Bärbel Mohr who advocates a similar philosophy of "cosmic ordering", and whom Noel Edmonds credits for his recent success. It works like this: if you need something, a washer-dryer, for example, as Mohr describes in a clip on YouTube, the universe will send you a washer-dryer.

Ehrenreich cites an article in the Los Angeles Times in which the reporter told how her sister, after having seen the film about The Secret , came to visit and "plopped a hand-tooled leather satchel on my piano bench and said, 'See the beautiful bag I manifested for myself?'" And the secret to this mystical provisioning? She had put it on her Amex card.

But, then, there's plenty of daft self-help out there, not to mention pseudo-scientific dangerous self-help too. I finally get around to picking up a copy of Louise Hay's You Can Heal Your Life , the book that has sold 35m copies, that I've seen people fawning over all weekend, and I flick through the pages. There's all the stuff I was expecting, What We Give Out, We Get Back, and The Only Thing We Are Ever Dealing With Is A Thought, And a Thought Can Be Changed. But then I find We Create Every So-Called Illness In Our Body. What? I flick to the back, where there's a whole list of these so-called illnesses with their so-called causes: acne is a result of "not accepting the self", arthritis stems from "feeling unloved", and the "probable cause" of Aids is "feeling defenceless and hopeless".

When I talk to Ben Goldacre, the NHS doctor who writes the Guardian 's Bad Science column and bestseller of the same name, he says there is evidence that beliefs and expectations can impact on your health, but that self-help is a "pretty seedy world" where writers often overdramatise these findings, and cherry-pick the evidence. He couldn't comment on the individual authors because "I would literally rather slam my cock in the door than read any more of these books".

But back to the people I meet. The ones who aren't into fairies or unicorns, who are simply trying to lead better, more productive, more thoughtful lives. Whom self-help has genuinely helped. Like Stephen Titterington, a 28-year-old chartered accountant from County Antrim, as nice and normal a young man as you could hope to meet. He credits the books for helping him pass his exams. "I'm not the most confident of people, and I had an overwhelming amount of work. In my firm, people are being sacked left, right and centre, we've been really badly hit. If I hadn't found this, I'd have let the fear take over."

Or, the woman who, when I ask how she got interested in the subject, hesitates and then launches into a sudden, heart-felt testimonial. "My husband ran off with a woman 20 years younger than me and left me with twin girls aged two-and-a-half, a seven-year-old, a 14-year-old and a 19-year-old. And you know what? If I hadn't picked up those books, I don't know what would have happened to me. It absolutely inspired me. You can get bitter or you can get better. And I've got better. He has my absolute blessing. I forgave him. I used to believe that I was a dumped woman. But I honestly believe now that he set me free."

Where's the harm in that? I ask Oliver James. It's just about taking responsibility, and changing how you look at things, and improving your life. "In my opinion it's extremely harmful. This is the story that selfish capitalism wants us to believe. That it's our fault as individuals that this fantastically big fuck-up in society happened, which Reagan and Thatcher caused, and which did not happen in mainland continental Europe. We have twice the level of mental illness as mainland Europe and yet this garbage encourages people to blame themselves and take responsibility, which is just a fucking joke. It makes me furious. It's very convenient to neo-liberals - meanwhile people like Philip Green have got massively richer while his employees read this crap and he nips off to Monaco in his £1.2bn corporate jet."

In Ehrenreich's eyes, it's self-help that has caused the mess we're in. And in James's eyes it's what will keep us there. It's just another opiate for the people. An Elastoplast, a form of textual Prozac, a device that masks symptoms but doesn't deal with our fundamental problems, either at a personal level or a societal one. It depoliticises us and reinforces a status quo in which we worker drones are kept just fit enough to do capitalism's bidding.

Maybe they're right. But it seems to deny, or underplay, the appetite that exists for some sort of help or guidance in dealing with the difficult business that can be life. Who couldn't do with a little extra help sometimes? But where are you supposed to get it from?

De Botton is at pains to point out that he doesn't support or condone the self-help genre, "because it's so disappointing and it could be done so much better", but he admits to the human need it shows up. "When people deny that we humans need such simple food, it's rather like somebody saying they don't need a cuddle."

Books should change your life. It's what writers have believed for centuries. Read Tolstoy. Read George Eliot. Read How to Win Friends and Influence People, if you must . Don't, and please just take my word on this, read The Secret .