We all know that "fat-shaming" is wrong. No one should be disadvantaged or ridiculed for their weight. In recent years, high-profile cases of fattism, from Karl Lagerfeld to Abercrombie & Fitch, have caused public outrage. Positive progress, certainly. But what about the flipside: why is skinny-shaming OK, if fat-shaming is not?

A few years ago when I worked in publishing, we'd gather for weekly commissioning meetings in the boardroom. There would be platters of pastries along the table. A senior colleague – a lovely woman in her 50s – would always urge me, loudly, to have a croissant. She would prod me in the side, in a friendly manner, and say: "Look, she's nothing but skin and bone!"

The fact that I was deeply anorexic and that she was overweight is irrelevant. She was drawing attention to my size in a way that would have been unacceptable had I done the same to her. I'm aware I'm skating on thin ice: what could be more irritating than a thin person describing another person as fat? And yet – for a moment – think about how we describe thinness: skinny, angular, emaciated, bony, skeletal, lollipop-head. These terms are batted about in the media quite casually, without the caution we must now use in our references to fat. I happen to find the term "skinny" offensive, but of course that's foolish. You're lucky to be thin, you think, rolling your eyes.

I've never been teased, or excluded, or called greedy. I'm not a "big girl" and I don't have curves. For 10 years, my problem was too little, rather than too much, food. What could I, recently recovered from anorexia, possibly know about fatness?

Well, I know the experience of feeling that one's private pain is on display on one's body, of being stared at, and feeling horribly conspicuous. I see clear parallels between fatness and thinness. I believe that out-of-control eating may work in the same way as out-of-control starving, as a defence mechanism against the world, a place to retreat when it all gets too much.

I write about this in my new book, The Ministry of Thin, and I've been shocked at the backlash. The plus-size sisterhood can be frightening. Among the messages I received (only from women, and mostly anonymous) I was called a skinny bitch, a body fascist, and a fat-nazi. I was informed that men "love something to grab on to", and that "curves" are sexier than skeletons.

And yet my book contains not a single word of criticism about larger-sized people. I employed the word "fat" in a literal sense, not as a term of abuse. I write about my own weight struggles, because I believe it's important to understand extreme emaciation – how your tailbone sticks out so you can barely sit on a wooden chair, how your limbs ache from lying in bed with no cushioning, how you bruise easily, and feel cold all the time.

It seems we can't have a rational debate about the reasons for, and the experience of, obesity – fat is still a feminist issue, and a fraught one at that. But I'm fed up with being judged for being physically disciplined, for watching what I eat, and for exercising five times a week. Other things a thin woman is not allowed to say: "it takes willpower to stay slim"; "of course it would be easier just to eat anything I wanted but I don't"; "yes, I'm often hungry mid-morning but I wait until lunchtime". Above all, a slim woman must never say: "I prefer being slim."

While researching my book, I spoke to a woman in the US who weighs 35st. She was friendly, open and happy to discuss her size. We got on like a house on fire, and I found her positivity refreshing, but 35st is an unsafe weight. I admire her ability to withstand societal pressures to look a certain way, but there is a level of self-deception here too: physical extremes, from very underweight to very overweight, are stressful for the body, and often mask mental turmoil too.

For me, facing up to the health consequences really helped. I wasted a decade struggling with the mental twists and turns of anorexia. It was only in the last few years, when I found a medical reason to recover – my fertility — that I made progress with weight gain. Could the same approach work for weight loss? If we reframed the debate around fat and accepted that it can be a form of disordered eating with physical consequences, we might start to get somewhere.

Emma Woolf is the author of An Apple a Day and The Ministry of Thin