When a friend came to my flat for coffee recently, I spent a panicked morning plumping cushions and checking all radiators for spare pants before realising I couldn’t offer her anything to put in her drink. In the fridge I had some almond milk, coconut yoghurt, and something made from “enriched soy oats” – but nothing that had been anywhere near an actual cow.

My diet isn’t exactly dairy free, as anyone who has seen me in action with a cheeseboard will know. I don’t have a good reason for avoiding milk, just a lazy, unchallenged assumption that it’s better for me when it’s made with nuts or oats and costs five times more than the standard sort. However, the Food Standards Agency has found that, among young people, a deliberate avoidance of milk is becoming common: 7.82% of 16- to 24-year-olds now believe they are allergic to cow’s milk compared to 1.36% of people aged over 75.

According to the British Nutrition Foundation, only 5% of people are thought to have a genuine problem with digesting lactose, the sugar found in milk. But a number of hugely successful food and wellness bloggers, including “Deliciously” Ella Woodward and Madeleine Shaw, suggest avoiding milk and trying dairy alternatives. It’s thought that they are a big part of this trend, with their young followers so keen to copy their idols that milk consumption is now plummeting. It’s true that this attitude to milk has become part of the cultural ether. I don’t follow any of these food bloggers closely, yet their message has got through – subconsciously I’ve started to believe that drinking cow’s milk is worse than smoking.

It’s easy to laugh off this phenomenon, and see it as an affectation, but I think it’s worrying that so many young people are excluding milk from their diets. When we become fixated on excluding certain foods from our diets, we damage our brains as well as our bodies. A healthy diet features everything in moderation, and there’s a place for broccoli and brownies. The culture that is causing us to avoid milk is encouraging us to be frightened of food. During my teens, I struggled with anorexia, and I frequently found excuses not to eat meals, claiming I thought I had an allergy or an intolerance.

Cutting out something specific might not cause too many problems at first, but it’s dangerous when it leads to further exclusions. When I was barely eating, I’d defend my choices by telling worried friends and family members that I was just trying to take better care of myself by eating more vegetables and avoiding high-calorie snacks. I’m reminded of this when I read about juice cleanses – a “cleanse” has an extreme, almost spiritual implication, but there are growing concerns that these cleanses are unhealthy and promote disordered eating. We know that severely restricted diets can lead to anorexia, but they also contribute to another headline issue – obesity.

A Harvard study in 2015 found that increasing the variety of foods we eat is associated with a lower risk of being overweight. In developed countries, obesity rates have quadrupled since 1980. Food fads can flourish in countries that have a relatively wealthy population who can afford to be choosy about their dietary choices. It isn’t surprising that the over-75s aren’t worried about whether or not to drink milk, as they have grown up with far fewer food options, and a sense that they are expected to eat everything that’s put in front of them.

Reading and writing about different ways to eat can be one of life’s biggest pleasures. It’s almost as enjoyable as food itself. Being open to new foods and culinary ideas can open your mind and enhance your diet. I admit I’ve enjoyed having the chance over the last few years to try cashew milk, courgetti noodles and cauliflower mash, some of the most ubiquitous dishes to come out of the wellness blogger boom. However, this is only valid if we’re adding to our menus instead of replacing staples. The milk-free trend isn’t making us healthier or happier. It’s encouraging us to be frightened of our food, and to give it too much power. Eating nothing but spiralised vegetables is just as bad for our bodies as eating nothing but burgers. More importantly, it’s bad for our minds too. To paraphrase Oscar Wilde, we need everything in moderation, including moderation itself. I think it’s time to start putting milk in my coffee again.

• This article was amended on 12 April 2017. An earlier version said the FSA found that 46% of 16- to 24-year-olds now believe they are allergic to cow’s milk compared to 8% of people aged over 75. What the FSA found was that 17% in each of those age groups said they had an adverse reaction to certain foods, or avoided certain foods because of the adverse reaction they might cause – and that of those, 46% in the 16-24 age group and 8% in the over-75 age group said they had an adverse reaction to cow’s milk and cow’s milk products.