There is too much of a guilt culture around parenting as new mothers and fathers are made to feel judged about issues such as breastfeeding and returning to work, the Liberal Democrat equalities minister has said.

Jo Swinson, who gave birth to a boy, Andrew, the week before Christmas, said new parents should do what instinctively feels right and stop worrying if they are not able to follow official guidance to the letter.

Speaking shortly before she went on maternity leave, she said parenthood was difficult enough without the societal pressure about how it should be done. "There is, I think, far too much guilt generally in society around parenthood, about whether or not you breastfeed or whether or not you bottle feed. We know the evidence is very strong in favour of breastfeeding and the benefits of that, but it shouldn't mean we make people feel bad if they can't do that for some reason.

"The whole stay at home or go back to work [debate] … mothers and fathers – particularly mothers, perhaps – end up feeling judged and guilty about decisions.

"I think we need to get away from this guilt culture in terms of judging other people's decisions as parents. It is difficult enough and it is not necessarily for others to judge."

Official Department of Health guidance based on World Health Organisation advice recommends that mothers should exclusively breastfeed for at least six months, but a 2010 government report found that only about 2% of mothers followed this advice. Despite evidence that breastfeeding is better for babies, promoting development and reducing the risk of disease, some groups, including researchers at Aberdeen University, have raised concerns that the guidance is putting too much pressure on new mothers.

Last month, a survey by the Care Quality Commission found 41% of new mothers in the UK said they felt inadequately supported in feeding their baby. One in seven said the perceived pressure to breastfeed prompted feelings of isolation and guilt.

About three-quarters of mothers start breastfeeding but only about half are still doing so at their baby's six-to-eight-week check-up – some of the lowest rates in Europe.

Research by Unicef has highlighted the significant health benefits of breastfeeding for longer, reducing cases of breast cancer in mothers, and gastroenteritis, respiratory illness, ear infections and the life-threatening condition necrotising enterocolitis in babies.

A spokesman for the children's charity said its most recent report argued there should be more support for the many women who want to continue breastfeeding, as it would be good for public health and save the NHS a large amount of money every year.

Swinson said the WHO guidance covered many countries where the alternative to breast milk would involve water that was very unsafe. "Certainly I know when my sister went through this recently, some of the advice she got perhaps more informally from health professionals, is that if your baby at four-and-a-half or five months is very interested in trying solid food, don't hold back and go with what your baby is telling you," she said.

"I think it is useful to have guidance and advice but you also have to be driven by practicality about what your baby is like. Is that necessarily a massive problem that someone should feel guilty about? I should say not.

"It's not to say people should disregard medical advice … but babies do develop slightly at different rates. Guidance is useful but we shouldn't be making people feel miserable and guilty about their parenting and the way that we do it."

Swinson, who is married to Duncan Hames, a fellow Lib Dem MP, said she had "a huge amount of support and goodwill" from colleagues in parliament during her pregnancy. While she is away, her ministerial work on employment relations, consumer affairs, women and equalities is being covered by Jenny Willott, one of her Liberal Democrat colleagues.

However, Swinson criticised some of the institutional difficulties of having a baby while being an MP.

"I think it's been lovely the way people have been really supportive in parliament of my pregnancy," she said. "[But] I think some of the structures of the institutions of the House of Commons probably don't make it as easy as it could be, in particular that you don't get maternity cover. As a minister, I get cover for my work … but nobody else will be being the MP for East Dunbartonshire."

Swinson said her staff would do a fantastic job of looking after her constituents but other countries allow a replacement MP to stand in.

"I don't think it's impossible or insurmountable but I don't think there's any job that's particularly easy to have a baby when you're in it," she said. "There's always going to be a lot of challenges and there's plenty of people who have jobs with even more difficult challenges, like people who are self-employed and running a business."

She condemned the "bizarre" ban on walking through the House of Commons voting lobby while holding a baby. "I hardly think it would be too much of a disruption," she said. "You can take a sword through there but you can't a baby.

"There has been a change that women who are breastfeeding can be nodded through. But I think when you are perfectly capable of walking through the lobby holding a small baby, I think there would be a better way of just allowing that. But parliament moves but slowly."

Swinson announced the news about the birth of her son on Twitter on Christmas Eve. "Duncan & I delighted to welcome our son Andrew Lennox Marshall Hames to the world on Sun 22 Dec, weight 7lb 10oz, all tired but v.happy!" she wrote.

Hamesadded his thanks to the "wonderful team of midwives who helped deliver our son".

"Really proud of Jo & am enjoying getting to know Andrew," he tweeted. He has stepped aside as a parliamentary aide to Nick Clegg, the deputy prime minister, to spend more time with their son.

• This article was amended on 6 January 2014 to correct an editing error. About three-quarters, not four in 10, mothers start breastfeeding in England.