Senior Conservatives have dismissed the idea of any attempt to unseat Rishi Sunak, even after two disastrous byelection defeats, but warned that the prime minister might instead be facing the equally grim prospect of a party on the verge of giving up. One Tory MP who campaigned in Tamworth, where the Conservatives lost to Labour by just 1,300 votes, said they were dismayed at the lack of top-level support, with one cabinet minister calling off a campaign trip the day they were meant to arrive. “It’s very frustrating, and it’s dispiriting for the staff,” the MP said. “If the ministers won’t turn up then the MPs don’t, and then the activists lose heart. It all starts from the top – even the PM wasn’t exactly visible. “We could have won Tamworth but our campaigning just wasn’t up to strength. Of course, there were national factors but on polling day it comes down to raw power on the doorstep, and Labour totally outgunned us. They were so much more visible.” Insiders said that such vignettes added to the impression delivered by the recent Conservative conference, of a party not so much gearing up for an election fight as preparing the ground for a succession battle once it is defeated. After the gathering in Manchester was marked by various ideological factions trying to push the party in different directions on areas such as tax cuts and culture wars, defeat in Tamworth and Mid Bedfordshire has seemingly reinvigorated all sides. With Liz Truss due to set out her own “alternative budget” a week before Jeremy Hunt’s autumn statement, pressure is likely to increase on the chancellor to make major cuts to income and inheritance taxes. Some MPs, however, believe this would be a big mistake. “We can’t head back into the Poundland Thatcherism of Truss,” said one. “MPs are obviously jittery. There is no enthusiasm for Labour but also an absence of any apparent strategy from the government. If we get it wrong, we risk turning a challenging situation into a disaster.” Another backbencher argued that Sunak’s strategy of focusing on cutting inflation rather than taxes was the right one. “The parliamentary party needs to unite behind our excellent prime minister,” they said. But another MP said Tory voters stayed at home during the byelections “probably because they don’t believe the government is delivering Conservative governance”. They added: “That needs to change rapidly, especially in the area of taxation – all eyes are now on Jeremy Hunt’s autumn statement.” Danny Kruger, the Devizes MP and co-chair of the right-leaning New Conservatives group, said Sunak needed to be “more coherent, more robust and braver” on areas such as transgender rights and migration, as well as tax cuts. “I really do believe that the PM wants to do the right thing in this, I think he is constrained by opinion in Whitehall, in the civil service and to a degree on our benches as well,” Kruger told the BBC. Some Tory figures are less openly supportive of Sunak, even obliquely. David Frost, the Brexit negotiator-turned peer, called the byelection results “extremely bad”, adding: “These results show that the national polls are broadly correct and that a strategy of denial is unlikely to work.” Nonetheless, there appears to be almost no appetite to try to unseat Sunak and install what would be a sixth Conservative prime minister in the past seven years. Miriam Cates, who co-chairs the New Conservatives with Kruger, told the Guardian it would be “crazy to consider leadership change”, adding: “But we need a king’s speech full of truly Conservative legislation that will appeal to our voters, voters who are currently minded to stay at home.” Even those loyal to Truss and Boris Johnson accept that Sunak will lead the Conservatives into the election. “No one is seriously thinking about [removing Sunak],” an ally of Truss said. “It would be preposterous.” That does, however, leave the party in something of a bind, beset by warring factions and notably unenthused by Sunak’s self-styled reinvention as the man to transform British politics. The MP who campaigned in Tamworth noted the sheer numbers Labour could call on, recalling the sight of a shadow minister knocking on doors surrounded by a “rugby scrum” of activists. The depleted Tory ranks were, in contrast, “a big worry”, they said: “Winning elections is a matter of muscle memory as much as anything, and if you lose that it’s hard to get it back.”