Boris Johnson is under increasing pressure to take immediate disciplinary action against the Conservative MP accused of watching pornography in the House of Commons. The chief whip issued a statement on Wednesday suggesting the matter should be referred to parliament’s Independent Complaints and Grievance Scheme (ICGS), which deals with sexual harassment and other disciplinary matters. But senior Tories questioned why he had not taken action directly against the MP, whose alleged behaviour was witnessed by two female colleagues in recent months. The identity of the MP is known to party whips, it is understood, but sources denied suggestions the MP was himself a whip. One cabinet minister described the situation as “madness all round”, while Caroline Nokes, the chair of the women and equalities committee, said: “I hope the chief withdraws the whip. In any other workplace they would be on gardening leave.” Chris Heaton-Harris, the chief whip, was present at a meeting of the “2022 committee” – a group of Tory women – earlier this week, when a series of incidents, including the MP watching pornography, were described. On Thursday the Labour leader, Keir Starmer, called on Johnson to bypass the potentially lengthy ICGS process, and discipline the MP directly. “This is an unusual case because the Tory party knows who this individual is. I think that they should deal with it, and deal with it sooner rather than later … They know who this is. They should take action now,” he said. Misogyny and sexism in parliament have come under close scrutiny in recent days, after a widely criticised Mail on Sunday story claimed Tory MPs believed Labour’s deputy leader, Angela Rayner, crossed and uncrossed her legs to distract Johnson at prime minister’s questions. Asked whether the MP accused of watching pornography should lose the whip, the prime minister insisted on Thursday that the case should be investigated independently. It was “obviously unacceptable for anybody to be doing that kind of thing in the workplace” but “what needs to happen now is that the proper procedures need to be gone through, the independent complaints and grievances procedure needs to be activated and we need to get to understand the facts,” Johnson said. A spokesperson for the whips’ office said: “This is the best way of handling this, because they can look at everything fairly, independently. This is the best process that we could have in place to deal with this matter.” At least five cabinet ministers have expressed alarm privately or publicly about the reports. Earlier in the day, the attorney general, Suella Braverman, said some MPs behaved like “animals”, and talked about a “coarsening” of behaviour in parliament and elsewhere. “I think this is something that does happen in many, if not all, workplaces, where a small minority of men – and it is men – are behaving in an unacceptable way,” she told BBC Radio 4’s Woman’s Hour. “I think that’s actually a more worrying symptom of our society and our culture. How have we got to a place in our society where watching pornography on the tube, in public, in front of children, in parliament, in the workplace has become somehow normalised for some people?” Nadine Dorries, the culture secretary, told the BBC: “I’ve been around a while and I think it’s actually getting worse [for female MPs] … I always thought that, get more women in … things will get better, but sadly I don’t see that happening at the moment. I think we just need to have a majority of women maybe. So it’s just disappointing.” Sajid Javid, the health secretary, said: “There have been several instances of inappropriate and sexist behaviour recently in parliament. It is shameful. Parliamentarians are public servants and it is their duty to set an example of character and integrity. The culture of Westminster needs to change.”