The defence secretary, Ben Wallace, has fuelled speculation of an imminent cabinet reshuffle, suggesting one could come in the next few days. In his first interview since announcing he planned to step back from frontline politics, Wallace said Rishi Sunak would inevitably have to reorganise his top team. Government sources said planning for a reshuffle was under way to allow the prime minister to “reset the narrative” if the Conservatives lose three byelections on Thursday. Tory MPs are braced for the party to lose two strongholds – Selby and Ainsty in North Yorkshire and Somerton and Frome in south-west England – as well as Boris Johnson’s old seat of Uxbridge and South Ruislip. Some hope a campaign against the London mayor Sadiq Khan’s ultra-low emissions zone could encourage Tory voters in west London to turn out in droves, but No 10 is said to be planning for its worst-case scenario. Wallace, who announced at the weekend that he would step down as defence secretary in the next reshuffle and as an MP at the forthcoming general election, relayed conversations that suggested a reshuffle this week was possible. Asked if he was still in charge of defence, Wallace said he still was “today”.. He said: “There is a rumour that there is a reshuffle tomorrow or the day after.” Pushed further at the Future of Britain conference organised by the Tony Blair Institute on Tuesday, Wallace told the audience: “It depends who you talk to at the top of government. I spoke to one very senior member of the civil service who said it is definitely happening this week. I spoke to a very senior member of the government, a minister, who said it is not going to happen this week. But it is going to happen, right? Either this week or the first week of September.” One government source said they had been told to prepare for a reshuffle on Friday. They said No 10 had asked for resources “only used during reshuffles”. A Tory frontbencher said a reshuffle this week appeared a growing possibility. They said such a move would allow Sunak to “reset the narrative” if the Tories lost all three byelections. The MP said conducting the reshuffle imminently instead of in September would allow new ministers to bed in over the summer, while those who were sacked or demoted would have time to “sulk on a beach”. There were furthers signs across Whitehall that a reshuffle could be imminent. An official said a government announcement they had expected on Friday had been moved, “presumably to clear the decks”. Sunak will have to fill the hole caused by Wallace stepping back, with Tom Tugendhat, currently the security minister, tipped as a possible contender for the defence role. Anne-Marie Trevelyan, the Indo-Pacific minister in the Foreign Office, said “it would be a privilege to serve” as defence secretary and she had “every confidence” that Sunak would choose the right person. Insiders suggested there would be limited changes to the cabinet, with only a few top ministers such as the environment secretary, Thérèse Coffey, potentially facing the chop. Instead, the bigger action is likely to be lower down the chain at minister of state level, it has been suggested. Those who think Sunak should wait until September say moving this Friday would look “panicky”. They argue that he should follow the timeline of Boris Johnson and David Cameron and instead reorganise his top team in the weeks leading up to party conference, which will take place at the start of October.