Dominic Raab has sought to quell a growing backlash over his alleged inaction while Afghanistan was falling to the Taliban, as a Tory former foreign secretary said Raab had been “caught out” over his handling of the crisis. On Friday evening Raab’s position as foreign secretary appeared to remain safe despite it emerging that he delegated a key call to begin the evacuation of Afghan interpreters to a junior minister while on holiday in Crete last week. In a statement, Raab insisted he had been “working tirelessly” to help as many people as possible. Tory MPs continued to privately call for him to quit and acknowledged that the row had been a disaster that had left the prime minister, Boris Johnson, “very weakened”. Speaking anonymously, a Tory former foreign secretary said Raab “should have been on the case”. Asked whether Raab would survive the political storm, they said: “In the old days it is difficult to see how he would have done. But now it seems possible to ignore conduct issues.” Raab’s conduct was also branded “indefensible” by a Tory former defence secretary, who argued that he should never have gone on holiday in the first place, let alone fail to return by the time of the first Cobra emergency meeting on Friday morning last week. Another former cabinet minister echoed the criticism, saying Raab’s judgment had been “extremely poor”. They added: “Frankly, sitting in a luxury five-star gaff in Crete while this was going on – and going there in the first place when it must have been perfectly obvious this was about to happen – reflects very badly on him. It is hugely disappointing.” They also said the absence of three of the most powerful civil servants – the permanent secretaries of the Home Office, Foreign Office and Ministry of Defence – was “extraordinary” during “a major international crisis which was so obviously about to happen”. Boris Johnson said on Friday afternoon that he “absolutely” had full confidence in Raab, adding: “The whole of the government has been working virtually round the clock, hitting the phones, doing what we can to sort it out and deal with a situation that has long been in gestation.” Johnson denied that Raab’s failure to make the call to his Afghan counterpart had delayed or inhibited anyone leaving the country. Raab’s position appeared to be secure when he released his first statement directly responding to the reports that emerged this week, which included a Whitehall source telling the Guardian that the foreign secretary “refused to be contacted” about some government business while abroad. Defying pressure to quit, the foreign secretary said the advice for him to call his Afghan counterpart reached his office at 6pm and was “overtaken by events”, meaning he delegated it to a junior minister – Zac Goldsmith – though he admitted the call ultimately never happened. Raab said he was too busy “prioritising security and capacity at the airport on the direct advice of the director and the director general overseeing the crisis response”. There appeared to be a concerted effort by some Tory MPs to rally publicly behind Raab in the wake of fierce briefings from his Conservative colleagues and those working in the Foreign Office. One of Raab’s defenders, Joy Morrissey, said the foreign secretary’s “armchair and Commons bench critics” were working tirelessly to apportion blame, and suggested they “start with the Taliban”. Helena Morrissey, the lead non-executive director at the Foreign Office, tweeted: “Something has quite clearly gone wrong here and that needs investigating, but in my direct experience, Dominic Raab has a very strong work ethic and attention to detail.” A senior Tory MP told the Guardian that Raab had “dropped the ball” and should quit, adding that “there isn’t really an excuse” for what he had done – though they predicted that Johnson would resist sacking him, instead piling pressure on the prime minister to bring forward a reshuffle that some speculate will take place in September though may be pushed back into the new year. Another said: “Boris didn’t sack Priti [Patel, the home secretary] for bullying; he didn’t sack Matt [Hancock, the health secretary] for shagging and he’s not going to sack Raab for not taking a phone call.” Despite the apparent temporary reprieve for Raab, a government source said he was hoping for a new cabinet post anyway, preferring a meatier domestic brief where he could be more political, instead of his current role where he felt hamstrung by the diplomatic nature of his job. Other senior Tory MPs said Johnson had a “very fractious party” and a reshuffle was badly needed. One said: “It’s better than Theresa May’s administration – anything was. But it’s certainly not living up to the previously promised hype.”