Derren Brown has written a book on happiness, so it seems reasonable to assume he is something of an expert. He says he is happy, and judging by the bibliography, he has definitely read a lot on the subject. But I’m not sure the two of us agree on what happiness is. His book, Happy, advocates “contentment”, “tranquillity”, a “considered” life. It all feels so unexciting.

“It doesn’t sound very joyful, does it?” Brown agrees. “But would you press the button for consistently joyful, which is clearly not real or a bit mad or exhausting, or would you press the button for a robust sense of calmness and balance?” I think, in that situation, I would hunt around for a third button, the one marked “frequent spontaneous bursts of ecstasy” – there is something listless about the thought of all that calmness – but Brown sees balance as “a much more helpful aim. I think it’s what we want.”

Brown, 45, speaks with a moderated calm, as if a kind of emotion-detector has filtered out the peaks and troughs of pitch. “Yeah,” he giggles. “I am a bit like that.” He has what he calls “an avoidant personality”, by which he means that he’s “very good at avoiding anything that might feel stressful”. Some people might think it would be stressful to play Russian roulette, as Brown appeared to do on TV in 2003, but he says that’s “just the job. In real life, when I can avoid anything stressful, I do.”

Brown describes Happy as “a philosophical, anti-self-help self-help book”. To put it crudely, he decries the work of Rhonda Byrne – author of The Secret, which coaches readers to believe that positive thinking will deliver what they want – and aligns himself instead with Seneca and Stoicism. “It is not events out there that cause our problems, but our reactions to them,” Brown writes. He is not himself a Stoic. “Not with a capital ‘S’” anyway. But Stoicism “was something I was living as I was writing”.

He wrote Happy over three years, hauling a suitcase of philosophy books along with him on tour. Writing was therapeutic, not least because during that time he “broke up with a partner of eight years”. He and Marc, a designer, remain friends. “It would have been easy for that to have slipped into a lot of anger with each other,” Brown says. Instead, with the Stoics in his suitcase, and advice on how to apply them clattering on to his keyboard as he typed, he became aware “that so much of it is about the story that you tell yourself, being in the moment with it, and not trying to control things you can’t control. I became very aware that the story I was telling other people about it was going to make a big difference to how I felt about it, and if I was going around looking for sympathy, saying ‘This awful thing happened, this big tragedy … ’, then it would start to feel like that.”

So the next day, Brown texted Marc; it was Marc who had ended things. “I said: ‘It’s silly that we’re feeling terrible about this. What we’re doing is a good thing. So we can feel fine about it.”

The next day? I ask, incredulous. After eight years, it took only 24 hours to make peace with his decision? “You must be pretty … ”

“Callous. Unemotional,” he butts in with another giggle. His giggles are a bit like snickers; there is something of the schoolboy about him, his delicate features, his back-of-the-class quips. (When asked how he takes his coffee, he replies, “Orally.”)

Not all his friends were convinced when he told them: “‘It’s a bit sad but it’s fine.’ I was getting the tilted-head thing. But actually – actually – those things are fine. I realised that whatever that relationship is, it will continue in some form. And how I behave will affect what it becomes in the future. So my priority became not to introduce anger.”

And right there, just as the subtitle of the book says, more or less everything was absolutely fine. Brown has made a habit of appearing to disclose his magical methods while leaving the stupor of his audience intact, and I feel the same bafflement now. It is as if he took all his feelings about the breakup and – puff! – made them vanish. Or, at least, brought them under control, which perhaps amounts to the same. Just a different kind of magic.

I don’t see how any technique could provide so fast and thorough a salve to heartache. “Oh, with this breakup?” he asks brightly. “Obviously, horrible night. That thing of waking up …” He draws a veil, a magician’s handkerchief, over the rest of the sentence.

Given that the hallmark of his stage and TV work is to extend his control and obfuscate its limits by blending magic with hypnosis and psychological manipulation, it is tempting to think that, in Happy, Brown is simply turning these powers on himself, making himself the object of his own suggestion. But he resists the idea that “there is an easy line to draw from my world of performing to the book”. He will only concede that “they both come back to me”.

Certainly, Happy is a new direction. The bibliography runs to three pages. AC Grayling and Alain de Botton have provided enthusiastic quotes for the jacket. “Validating!” Brown grins. Despite his proclaimed fear of dinner parties, he has dined with both philosophers and found them “delightful”. He can see Happy sitting “very comfortably” in Waterstones’ “Smart Thinking” section. And if it doesn’t, well, he can always do what he did with previous books and move it.

“I used to do that a lot,” he says: to his irritation, bookshops kept filing Confessions of a Conjuror under Games & Puzzles. Given these habits, and the book’s talk of taking control of the stories we tell about ourselves, I wonder if Happy is Brown’s attempt to steer his own story along a more intellectual path. The word “magically” crops up repeatedly and colloquially in the book – seeing partners “in magical terms”, and so on – as if Brown wants to divest the word of its professional meaning, so he can pass as a philosopher. After all, his next show is called No Tricks, and really does contain no tricks, just Brown talking about happiness. Isn’t the book an attempt to re-frame his own story in this light?

“What I mean by the story we tell ourselves, it’s not so much having a broad … Maybe I do have a broad narrative. But it’s not in the sense of ‘I’m this guy, and I’m aiming here’,” he says. He has always claimed to lack ambition – to the extent that I’m wondering if the insistence is itself part of his story. He has no specific aims for his private life, either. He has a new partner now, but marriage is “one of those things that my avoidant personality very quickly shuns. Not avoid in the sense of ‘I don’t want to get married’, but ‘I don’t want to think about that now’.” Children? “Again, no strong urge.” He is having enough trouble with his puppy, Doodle.

I can’t help wondering what emotional vantage point Brown occupies in relation to his book – what great happiness or unhappiness he has known. Maybe his avoidant personality is at work again, because he says: “I don’t look back and find any time especially unhappy.” When he was living in Bristol, for instance, after graduating in law and German, he lived on housing benefit with a pet parrot. He was just starting out in magic, and remembers thinking, “Well, if I take a cross-section of my life at the moment, is everything as I’d like it to be? And it was. Everything felt fine and in the right place.”

This is surprising, given that he had yet to come out and was still trying to “cure” himself of homosexuality. He was “certainly insecure”, he says. “But I don’t know if that really made me unhappy.”

It is hard to square his emotional “cross-section” with other evaluations he has made of life in those days. For example, 2009’s Confessions of a Conjuror opens with a description of the self-loathing he felt while performing magic. It’s not as if his parents disapproved. (When he told his mum he was going to be a magician, not an international lawyer, she said, “Oh, that sounds much more fun.”) How does he explain the disjuncture?

“Ask any magician what they do and you will get the same, squirming answers,” he replies. “There’s something a bit embarrassing about saying you’re a magician. It immediately suggests all these horrendous cliches, let alone that you’re a grown-up doing a child’s job. I think the opening words of my last book were ‘I loathe myself’. It’s not how I feel now. It’s not how I am now in relation to me now. It’s part of any magician’s relationship to their own craft.”

But magic is just one element of the past for which Brown has expressed distaste. Tricks of the Mind, his 2006 book, begins with a vitriolic glance back at his Christian days – as a child growing up in Purley, south London, he attended Bible classes, and began to question his faith only during his university years, when an interest in magic and hypnosis took hold. “It’s the intellectual faint-heartedness that feels embarrassing,” he says. “I do always look back and feel faintly embarrassed by anything I’ve done in the past. I think that’s not a terrible thing, because if you don’t do that, how are you growing and moving forward?”

I worry that in a few years he will look back at his current incarnation – let’s call it the Derren Brown: Philosopher phase – and cringe.

“I’m sure I will, I’m sure I will,” he says, before finishing up with the words “... so I don’t think I will.” When I notice the contradiction later and email him, he explains that his “instinct changed” as he spoke. The latest book “comes from a richer place” than the others. Maybe it will withstand his habit of retrospective loathing. “I do naturally cringe a bit at most things I’ve said or done in the past … ” he says. “That has generally been the pattern.”

With his thin, calm voice and his scrupulous politeness, I am struggling to picture Brown overwhelmed by happiness. Is his tranquillity ever scuppered by uncontrollable happiness? “Oh God, totally. I don’t know if you do this as well? When you’re with your partner, I think, does everyone else sing and do the stupid voices and all that stuff that I do and always have done?” Crucially, he thinks this in the moment; his happiness is moderated by his awareness of it, and I wonder if that self-awareness is a necessary component of the happiness.

In fact, when I ask what makes him happy, his partner is not the first reply. It’s “writing in cafes, and painting” – creative pursuits that “get you out of your own head”. But, of course, they don’t really. They just make it OK to be in your head. His book is peppered with images of himself as author, sat in cafes in Norwich, Cardiff and Sheffield - small, subliminal suggestions, perhaps, that this is the authentic Brown, that the wizard behind the curtain is really a writer at a laptop. If neither painting nor writing nor photography are feasible, “I’ll start picking up litter or talk to someone who seems lost with a map, then take them where they want to go. It sounds truly, awfully sappy. It works, though.”

“Happiness,” Brown says, “is partly the story you tell yourself. Meaning doesn’t happen in the moment without some sort of before and after, and a story being drawn out of it.”

The funny thing is that he resists applying that “before and after” to his own experience. He insists he is happy, and at any moment in which he revisits the past, confirms that that was a happy time – despite the fact that he looks back with “self-loathing”. It seems a curious trick of perspective, that each moment should be fine until it has passed, and then, with the benefit of hindsight, becomes hateful or, at least, embarrassing. No wonder he chooses to live in the present. I am beginning to wonder if his stoicism is the necessary defence against the more painful effects of self-scrutiny.

“To approach [happiness] the other way, and see it as an absence of disturbances is helpful,” he says, and although his words have the same weightless equanimity as always, the idea feels poignant.

Towards the end of Happy, he writes of “realising that we are all alone, that no one is ever entirely right for us because we are all broken, and that we can only open our broken aloneness to that of another”. Happiness sounds like such a forlorn experience. “I don’t think it’s a sad thing,” he says. “It’s only sad if you tell yourself it’s sad. I think like anybody, I am going to be maddening and irritating and – yeah! – damaged goods. We all are, aren’t we? Absolutely. And that is fine.”

Well, I say, I’m looking forward to the next book. Despite his insistence that he doesn’t forward plan, I’m pretty sure there will be one.

“Sad,” he says. “I was thinking that the other day. Maybe the next book will be about sadness.”

• Happy: Why More or Less Everything is Absolutely Fine by Derren Brown (Bantam Press). To order for £16.40 (RRP £20) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p on online orders over £10. A £1.99 charge applies to telephone orders.