Keir Starmer once claimed Boris Johnson was “not a bad man”, just “unserious”. He is close to reappraising that now after a fortnight of rows and threats over the prime minister’s false accusation that Starmer failed to prosecute the paedophile Jimmy Savile during his time at the Crown Prosecution Service. That accusation – which the prime minister has “clarified” but never withdrawn – was a moment for many Conservative MPs. Though only a handful have spoken publicly, Starmer told the Guardian that dozens had been in touch with him privately to express distaste. “You can see the disquiet because as he drags people down by this way of acting.” Starmer said it was clear the public were taking a new look at Johnson’s character since his repeated obfuscations over Downing Street lockdown breaches. Has he changed his mind since saying in his Labour conference speech that Johnson was not bad? “I don’t know how people judge whether someone’s a bad person or not. What I do know is he’s not fit to run the country,” he said. “And it’s not surprising that more people think he should now resign than don’t. So that’s a marked change, obviously, in four or five months.” Labour has held a robust lead in the polls since mid-November though party chiefs remain cautious over how much Johnson’s “partygate” troubles have inflated the lead, with voters still repeating that they do not know what Starmer’s Labour really stands for. Methodology changes from some pollsters have recently shrunk Labour’s lead, taking more into account how voters are switching to “don’t know”. Lisa Nandy, the shadow levelling-up secretary, has driven home in recent weeks that the party needs to be far more visible presenting its own agenda. It is recess in parliament and Starmer is spending most of it in the north of England and the Midlands: Sunderland, Burnley, Erdington. On Tuesday, he was on the factory floor of the plastics manufacturer What More UK, giving an amused Nandy a detailed tour of the toolmaking machines two decades after his toolmaker father used similar ones. Labour has made big promises to kickstart a renewal of British manufacturing and Burnley is a stark illustration of the north-south divide in industry. Census data shows that for each high-skilled manufacturing job, there are five low-skilled ones. In Cambridge or Crawley, it’s one for one. A major local employer, Studio Retail, called in administrators last week, putting 1,000 jobs at risk. One of the factory bosses who showed Starmer around estimated it was about one in 30 jobs in the local area. “If this were another part of the country, there’d be a government taskforce,” he said. Burnley has been a hostile environment for Labour politicians recently – it fell to the Tories for the first time in 100 years in 2019 – but for much of his visit, Starmer was in a comfort zone of sorts. He told his family history at length to any worker who would listen as he roamed the factory. His second stop was meeting law students at Burnley college, where he was questioned on practical routes to the bar, and about his first legal case defending careless driving: “I lost.” At one point, he relaxed enough to declare to the ambitious students “the next leader of the Labour party could be in this room” and then realised Nandy was grinning beside him and laughed. About 60 wavering voters met Starmer on Tuesday evening in the college to talk about Labour’s “contract” – security, prosperity and respect – which they were tasked with analysing. The mood was generally bleak. Just two raised their hands when asked if the town would be better off in five years. On a table of 10 younger voters, none said they planned to stay in Burnley. “It’s not a viable option,” said one. Another said: “If you want to do well, you can’t do it Burnley. I’m never proud to say I’m from here.” Older voters describe the housing stock as “decimated” and express fears for a generation held back by Covid. There was anger, too, at politicians, but not overtly at Johnson. “We keep hearing lies, all the time, from politicians,” one woman said. Starmer said he acknowledged Labour had to transform more voters turning against Johnson into making a positive choice to vote for his party. His first reply has been a radical overhaul of his party, including divisive policies at party conference, a reshuffle that reinstated many Labour heavyweights sidelined in the Corbyn years, tough stances on antisemitism and unequivocal backing of Nato, with a sideswipe at Stop the War coalition where his predecessor holds a leading role. Activists and MPs on the party’s left have spoken of feeling betrayed at this uncompromising attitude. But Starmer said his main focus was winning back trust from seats like Burnley. “When you lose former Labour voters, then you have to look at the Labour party and say – what do we need to change in the Labour party? – and we’ve made those changes,” he said. “We’ve got more to do and every vote has to be earned. And that’s why we’re here.” Nandy, the MP for nearby Wigan, is cautious about how far Labour has advanced in this part of England. “We’ve got more work to do,” she said. “We have to show them we can be trusted with a very scarce amount of money people have.” But still too few voters had heard their policies, she said. “When I talk to people in Wigan, in Leigh, they have been positive about our offer but they didn’t know about it. That’s why you have to be here, on their territory. “Voters talk about energy bills but they don’t know our policy of a windfall tax on North Sea oil and gas producers. We’re doing something wrong when we’ve been saying nothing but that for the last three months and people haven’t heard it.”