My dad said it to me when I was seven years old and it stung like vinegar on a paper cut. Of all the things you throw at kids you never know which ones will stick. This one accidentally stuck. I’m not black enough. The phrase, the unblackness, was planted, and developed like an irksome bruise I only feel when I bend a certain way. It’s a scar tissue formed from acid poured into my wound after I was hit by the paternal truck of not-black-enoughness. An unexploded bomb that’s leaking mustard gas into my blood.

I’m fully grown now, but the comment still tinkles lightly on the piano of my mind. Whiteness is in my blood. Well, half of it. My mum’s never done an ancestry DNA test, but as a woman from north London who regularly burns in the British sun, it’s safe to assume she’s majority Caucasian. I’m mixed-raced and grew up in multicultural Brighton. When my parents split up, my dad stayed in Brixton and I was occasionally evacuated to him on school holidays. As a single parent, aware of her son’s racial identity, my mum rallied, and founded Mosaic, a support group for mixed-parentage families. Brighton is inherently liberal, but “How come your mum’s white, but you’re not?” was the power ballad of my childhood.

Luckily, each peer-to-peer micro-aggression that questioned my identity actually fortified my sense of self. Sorry to go a tad Oprah, but they were all learning experiences en route to the slightly too self-assured man I’ve become. I call it public school confidence, but without the fees. Of all the things I am – tall, loud, almost aggressively expressive – black is the one people see first. But I’ve never felt defined purely by my race. That might be what my dad was getting at.

I can’t truly shake the reality that I have lived a predominantly white life, with a white mother and stepdad and – in the past 20 years – a four-bedroom home in the countryside, a boat and a house in France. These shiny new things don’t tally with my mum getting chased down the street with me in a buggy as people screamed the N-word at her. I’m not steeped in the residue of my 1980s Brixton youth, nor is this quite the rags-to-riches My Fair Lady story.

As I gaze around me at the white middle-class paraphernalia, my amassed signifiers of success, I know I am culturally white. It’s a sticky, uncomfortable feeling, like the fur on unbrushed teeth, or a door that doesn’t close completely flush. That’s not to say there are no successful black people on the same page as me, but I haven’t lived out the reductive story of blocked opportunity so often associated with inner-city black men.

I wish there was a turning point, some clear moment I assimilated, but the whiteness and privilege were my climate and I just changed. Like any human, I adopted the values of the lives immediately adjacent to mine.

I’ve both succeeded and failed as a black person, and that’s a tricky thing to fully express without falling into tired stereotypes. I sometimes feel like a kind of poster boy for liberals, because I’m an ethnic some way up the ladder of successful capitalism. Ironically, my biggest fear is that my blackness has opened more doors than it has closed. That rankles me more than the missed opportunities. I joked in my teens that I’m catnip for middle-class women, and that’s somehow still a constant nag. Have I been championed by people who quietly knew they were doing something wholesome by supporting me? Not charitable exactly, just good.

Positive discrimination should be invisible to the naked eye, but racism in the UK has followed a similar path, evolving alongside endeavours for equality. People aren’t noticeably discriminatory; the least accepting people take the most umbrage to being called racist. But racism is thriving, invisible like carbon monoxide, inhaled through Brexit and exhaled in a constant hammering criticism that sees a duchess and her husband relinquish their titles.

We all want to be liberal. To love thy neighbours, even the ones we are only connected to by follows and likes. But we’re right-on splodges on a massively biased canvas. The system is skewed. Sometimes I feel as if I’m compensating or I’ve diluted my blackness like vodka in a cheap club.

I’m paranoid I’m the palatable black man, the kind of guy you party with, the kind who won’t kick up a fuss unless you make an explicitly racist joke (the London bubble keeps me pretty sheltered from those, to be honest). Have I gradually made myself more digestible, WD40ing myself for social lubrication because I don’t want to be the guy with a chip on his shoulder bringing everyone down? Is this a form of social survival, or just the self-levelling screed of living in a community? I guess it’s the same palatability that’s expected of women the world over, so I’ll put my tiny violin away for now.

In my book I explore the idea of not being a specific type of gay – not an otter, or a bear, or a daddy – in the same way David Hockney isn’t a painter associated with a specific movement. As the Hockney of gays, I wonder if there’s also any mileage in the type of black person I can be? Not explicitly my father’s type, but one of my own.

I’m Jamaican pigmentally, but too anxious about the island’s approach to homosexuality to get on a flight to Kingston. I’m sure I’d be safe as houses at a luxury resort, but that doesn’t scream finding yourself, does it? It’s hard to connect with your roots while sipping Mount Gay from a coconut by the pool. I’m Rastafarian in name (my initials deliberately spell RAS), but not versed in the religion at all. I love reggae, but is there anything whiter than reggae?

My personality is always on my sleeve, my emotions written all over my face. My eyebrows will tell you when I’m ecstatic. And it’s crystal clear when I’m displeased. I’m easy to read, like a Mills & Boon or the Sun. I’m, what you might call, a lot. I always want another bite of the cherry. Enough cherries to get diabetes. My dad is pure charisma, too. Very, very dryly funny and thoughtful. But even at five or six, I manufactured a lessness to appease him. Medical grade introversion. Less keen to dance. Less gobby. Less Raven Smith. Visiting my dad in London always came with a side order of a slight suppression of the self. A quieter, more subdued boy would return with his mother on the train.

Speaking of which, my dad once dreaded my hair and my mum spent the entire train journey back home painfully combing the wax out.

Suddenly feeling the low-level hum that he should dutifully be included in my nuptials, I went to see my dad just before I got married, after a five-year break. I could still feel the sting of his words as I entered the flat. Not black enough. His questioning of my blackness wasn’t strictly about race, it was a questioning of my relationship with him. He was the source of blackness in my life and he could feel me slipping away at seven. A great man, Windrushed here at 14. But his fathering style wasn’t the type that celebrated a high-spirited child who needed channelling rather than being dammed up. I was coursing away from him and he couldn’t stop the flow.

My father and I were, and still are, two circles of a Venn diagram that couldn’t work out how to overlap. I always felt like this man never really “got me”, that by some splice of DNA I came out of the womb attached umbilically to his sense of disappointment. I was never tough enough. I was never straight enough. I was never black enough. But the truth is, I was never close enough either. Circumstances separated us and misfired attempts at communication, with a focus on racial identity, kept us apart. You’re not black enough should have really been you’re not here enough. Or, you are enough. Or just, I love you.

Raven Smith’s Trivial Pursuits by Raven Smith is published by 4th Estate at £10.99. Order a copy for £9.23 from guardianbookshop.com