"The purpose of psychology is to give us a completely different idea of the things we know best," wrote the French philosopher Paul Valery. But we often resist: we imagine we already know ourselves and those closest to us perfectly well enough.

The history of psychology is littered with experiments in which ordinary people behave in bizarre, irrational, sometimes hilarious and occasionally terrifying ways. Yet who wants to believe they'd do so themselves? Sure, 65% of the participants in Stanley Milgram's infamous studies on obedience were willing to deliver what they thought were dangerous electric shocks to members of the public, just because a man in a lab coat urged them to do so - but you probably think you'd have been one of the hold-outs.

Other research indicates that people respond more positively to a romantic overture from a stranger when it occurs on a wobbly bridge, rather than solid ground; that they're likelier to trust a salesperson who's mirroring their physical movements; or that when they've accidentally bought non-refundable tickets to two clashing events, they'll usually decide to attend the more expensive one, not the one they expect to enjoy more.

But that's them, not us, right?

Look beneath the surface, though, and this confidence in our own self-knowledge and common sense soon gives way to insecurity; the ever-increasing market in books on self-improvement and happiness is testament to that. The shelves of British bookshops heave with good science and pseudoscience, marvellous advice and terrible advice, contributions from professional academics, Buddhist monks, therapists, hypnotists, philosophers, business gurus, novelists, film stars, and Noel Edmonds. Oh, and Rhonda Byrne, the Australian television producer whose book The Secret - a partly mystical and entirely ridiculous guide to obtaining sports cars and piles of cash by the power of thought alone - has become one of the bestselling books of all time.

It's easy to see the pop-psychology phenomenon as a depressing symptom of a sad, needy, alienated society: why are we so desperate for advice and consolation, and why can't we find it from friends and family? Besides, plenty of the factual claims you'll find between self-help covers are factually wrong, or unproven at best. For example, we don't use only 10% of our brain's capacity, as fist-pumping motivational coaches like to claim.

And yet a cynical dismissal of the entire sector misses something profoundly exhilarating about it all. We're lucky, for example, to be living in the era of "positive psychology", a movement among experimental psychologists to broaden the discipline's focus beyond disease - to investigate what's happening not just when things go wrong with the human mind, but when
things go right.

The online culture of "lifehacking" explores tips and tricks for maximising happiness and productivity, treating the mind like a children's chemistry set, ripe for tinkering. And the explosion in the popularity of books on happiness signals a refreshing shift towards honesty. All we've ever wanted is to feel happy; after all, that's what motivates hedge fund managers and aid workers alike. Perhaps we're just getting less secretive about it.

This two-part guide, free in the Guardian tomorrow and in the Observer on Sunday, offer entry into the fascinating world inside your own head, and the heads of those with whom you share a home and a workplace. They won't tell you how to think your way to a fortune, or how to develop irresistible charisma using only the power of affirmation.

But experiment after experiment has concluded that learning how the mind works is the best antidote to the irrational and self-defeating cognitive glitches to which we're prone: once you know how easily we succumb to groupthink, or foolishly worry about one-in-a-million aeroplane crashes rather than road accidents, or obey men wearing lab coats, it's easier not to do so.

Can you also use the information contained herein for more Machiavellian purposes: getting people to buy you things, fall in love with you, or otherwise do what you want? We make no such claims, and cannot be held responsible for the results.