Tory MP Sir Gerald Howarth called it wholly inappropriate, while Channel 4 presenter Cathy Newman said it hardly advanced the cause of feminism. Neither were talking about the latest cavortings in the Celebrity Big Brother house, but the suggestion by a government minister that she walk her newborn baby through a door to vote.

In an interview given in the run-up to Christmas before her baby was born, the equalities minister, Jo Swinson questioned the anomalies of a workplace where breastfed babies are allowed into an MP's office, but not in either House chamber or the division lobbies they pass through to vote. "I hardly think it would be too much of a disruption," she said. "You can take a sword through there but you can't a baby."

In saying so, Swinson proved yet again how useful it is to have an equalities minister, who by daring to give birth to a child while in office, is a living embodiment of her job title.

The outraged reaction of several Tory MPs, who obviously would rather not have these pesky Lib Dems in any sort of office let alone a cabinet one, is perhaps predictable. Howarth snorted that parliament was not a creche, a comment that failed to make any reference to the usual shenanigans during PMQs. But the opinion of Newman suggests that the idea of taking babies into work is one of those issues that the British find particularly annoying.

Under the law, which Swinson is now partly responsible for, employers must carry out a risk assessment to establish whether pregnant or breastfeeding women can continue to work. If they can, the employers have a legal obligation to provide suitable rest facilities. Parliament, unlike many employers, has a creche for MPs that is usually open until 6pm. So far, so good.

Yet Swinson, who is not allowed maternity cover for her duties as an MP (though she gets it as a minister), is simply suggesting that during these early months, while she's trying to sort out breastfeeding and sleep and carry out the job to which she has been appointed, she would quite like to be able to hold the baby – even when needing to vote. After all, it happens in the European parliament, where Licia Ronzulli brings her daughter in to work so much newspapers have devoted full picture galleries to her development.

Newman, the mother of two girls, makes the point that bringing a baby into work is disruptive to you and annoying to others. "Juggling life and work is hard enough without having a baby in the office competing with your boss for your time and attention," she writes. Yet doesn't it depend on the job, the parent and the circumstances?

I can think of plenty of times when taking a baby into the office is awkward, if not downright embarrassing. My first boss at the Guardian came to see me while I was on leave and had a meeting while I sat with a hungry newborn squawking at my breast. Both Newman and I have jobs that allow us occasionally to work from home, where our children can be close. Like her, I find them a bit too distracting to be around while I'm trying to work. But others don't. They are probably far more productive than me so why should we stop them?

This isn't just about those of us lucky enough to have such flexibility either. I had a cleaner during my first maternity leave who had a baby about the same time as me and would bring the little girl to our apartment while she cleaned. I felt so embarrassed about this inequality – not realising that maternity leave could turn you into a sort of Lady Muck – that I took to leaving her alone to clean while I walked the streets of Manhattan. It never even crossed my mind to tell her she couldn't bring the baby to work.

Given that Swinson was partly responsible for the act that will allow both parents to share maternity/paternity leave that comes into force next year, I'd like to know where her husband, fellow Lib Dem MP Duncan Hames, stands on the issue. In a prescient interview before the birth, she talked of the huge expectations of women as mothers, not just during breastfeeding but long after. "It's an incredibly difficult thing to juggle parenthood, and I don't think it's easy for anyone, and everybody has to find a way of making it work, and that's just what we are embarking on."

It made me wonder what the reaction would be if Hames asked for the rules to be bent so he could walk little Andrew through the lobby division. Would we consider it a blow for feminism? Or an outrageous demand for a nanny state? If the answer is yes to either of those questions, then maybe the equality that Swinson's job is meant to fight for has already been achieved. Men can't breastfeed or give birth but they can hold a baby while voting can't they? That would be true equality.

• This article was amended on 7 January 2014 to clarify that Jo Swinson gave the interview before the birth of her baby.