Removing Boris Johnson from Downing Street would lead to "instability and uncertainty" in the country, according to the Conservative Party"s chairman. The PM has come under increasing pressure from MPs after being fined by the police for attending a party in No 10 during the first lockdown. Some senior Tories have now joined opposition calls for Johnson to go. But Oliver Dowden said changing leader right now "would not be in the national interest". Speaking to the BBC"s Sunday Morning program, the party chairman said the UK faced "unparalleled challenges" over national security and energy supplies, and it was right for the prime minister to focus on that. But Labour leader Sir Keir Starmer said the PM"s authority was "shot through", and the so-called partygate scandal was stopping Parliament discussing issues like the cost of living crisis. The SNP"s Westminster leader, Ian Blackford, also said Johnson now had "no moral authority" to lead after being fined for breaking COVID laws. Last week, Johnson — along with his wife and Chancellor Rishi Sunak — was fined by police for attending a birthday party thrown in his honor in the Cabinet Room in June 2020. It is one of more than 50 fixed penalty notices handed out by the Metropolitan Police since it began its investigation into the law-breaking parties in Downing Street and across Whitehall. And it was the first time a sitting prime minister had been found to have broken the law. The PM repeatedly said sorry to Parliament on Tuesday, saying while he did not think he was going against regulations at the time, he accepted the force"s decision and wanted to now focus on getting on with the job. But Labour deemed his apology "a joke", and along with resignation calls from the SNP and Liberal Democrats, a number of senior Tories said it was time for Johnson to go. Former Brexit Minister Steve Baker told the Commons "the gig is up", and warned in the Daily Telegraph that partygate had been "a disaster", adding: "I fear we will reap the whirlwind on polling day." On Thursday, MPs also voted for an investigation by a parliamentary committee to take place over whether the prime minister misled the House over his statements on No 10 parties. Knowingly misleading MPs is a resigning offence under the ministerial code. Liberal Democrat deputy leader Daisy Cooper said there was "a point of principle" in holding him to account, telling the BBC: "If you allow a prime minister to mislead Parliament without any repercussions, we end up in a very dangerous place." Dowden defended his boss, telling the BBC: "I understand how people feel really hurt and angry about this, and I think the prime minister acknowledged that properly in what he said — he gave a full and unreserved apology. "But I think you do need to balance against that first of all the really good things he has done as prime minister, whether that"s delivering the vaccine program, getting Brexit done, or the actions in Ukraine. "But also the challenges we face right now, whether that"s national security, or whether that"s energy security — these unparalleled challenges. "And I actually think that instability and uncertainty caused by a change in leadership would not be in the national interest." But Labour"s leader said Johnson was to blame for the focus on parties. Sir Keir told the BBC: "Why are we talking about this? Answer: Because the prime minister has broken the rules he made, and been fined by the police for doing it. "No prime minister in the history of our country has even been in that position before. So he"s brought this on himself." He added: "His moral authority, his authority to lead, is shot through and his own side have now had enough of defending him." — BBC