Michael Gove has suggested disciplinary action could yet be taken against the Conservative backbencher Desmond Swayne after he refused to apologise over controversial remarks to vaccine sceptics. However, the Cabinet Office minister said he was sure that Swayne, a former minister, would say sorry over comments made in November saying that Covid figures were being “manipulated” and that intensive care occupancy levels and excess death rates were not out of step for the time of year. Asked why Swayne had not faced disciplinary action, Gove told reporters: “Watch this space.” He added: “I can’t pre-empt any decision-making, but I think it’s important that the processes that the Conservative party has are in place in order to make sure, as I’m sure he will, that Sir Desmond apologises and recognises he made a mistake.” Swayne has claimed that criticism of him over the matter – and of another interview he gave to Del Bigtree, an anti-vaxxer and producer on Vaxxed, a 2016 film directed by the disgraced former physician Andrew Wakefield – was “disinformation that they’re spreading by lumping us all together when people have certainly legitimate points of view in a free society”. During the interview with anti-lockdown group Save our Rights UK – which this week had videos taken down by Facebook for breaching its “Covid-19 misinformation policies” – Swayne told the controversial protest group to “persist” in their endeavours. “It seems to be a manageable risk, particularly given the way figures appear to have been manipulated,” he said. “We’re told there is a deathly, deadly pandemic proceeding at the moment. “That is difficult to reconcile with ICUs [intensive care units] actually operating at typical occupation levels for the time of year and us bouncing round at the typical level of deaths for the time of year.” It is understood the former minister has been spoken to by the Conservative chief whip, Mark Spencer, and has been told to meet government scientific advisers, but has so far avoided more serious punishment. In a Sky News interview on Thursday, Swayne said the situation had “changed completely with the new variant”, but that his remarks were “perfectly legitimate at the time”. He also told TalkRadio: “We are getting very close to thought crime here, aren’t we?” Challenged over the apparent risk that he could be endowing respectability to fringe positions, Swayne told the Guardian: “There is a danger of that and I have on occasion responded to a request for an interview and failed to have found out about exactly who they were interviewed by and what their line was and found myself uncomfortable in the interview.” However, he added that he was “not sure” if he would count Save Our Rights UK in that category. Labour’s deputy leader, Angela Rayner, said that “after the Conservatives have refused to act”, the prime minister must “intervene urgently, condemn these comments and take action”. Gove had said on Thursday that Swayne was “out of order” and must apologise, adding: “I work with Sir Desmond, I have great affection for him but I’m afraid here he is completely out of order. I would hope that he issues a full and complete retraction and apology for what he said – it’s unacceptable.” Priti Patel, the home secretary, later also called on him to apologise and retract his comments, which she said were “thoroughly wrong”. Swayne, 64, previously refused to apologise over his wearing of blackface at a party. He said that although he would not do it again, that was only because the makeup was difficult to remove. “The reason I wouldn’t do it again is because you have to go to some trouble to get it off. It’s not a non-trivial exercise,” he wrote.