We may live in a largely secular society, but the Protestant work ethic is still alive and strong. The “lazy” and “entitled” millennials, we have been told, are workshy and self-indulgent. They spend too much and save too little – behaviour that is not only harming their future prospects, but those of the world economy. We should have the grit of our elders, apparently – who weren’t scared to suffer some hardship with the promise of a better life ahead. Except they too are coming under criticism for enjoying the life that they struggled to earn. According to the UK chancellor, Jeremy Hunt, it is time for the over-50s to put away their golf clubs and start contributing to the economy again. The gospel of self-restraint clearly runs very deep in the cultural psyche and, until recently, psychological research had seemed to confirm that delayed gratification was indeed the secret to long-term success. Yet some recent research has come to question these ideas. While moderate levels of willpower are almost certainly beneficial, people who attempt to avoid all kinds of indulgences are neither happier, nor healthier. They are not even more successful at achieving their goals. By embracing rather than shunning our short-term desires, and knowing when and how to indulge them, we may enjoy greater wellbeing with no cost to our productivity. The long and the short of it By definition, delayed gratification is the idea that putting off temporary pleasure in the moment will lead to greater contentment once we have met our longer-term targets. The research may be best represented by the famous “marshmallow test”, in which children were asked to resist eating one marshmallow immediately with the promise of enjoying two marshmallows a quarter of an hour later. Years later, those who had succeeded in the marshmallow test got better grades at school and progressed more quickly in their careers. The finding proved to be so influential that it inspired education programmes devoted to building character; the idea even infiltrated Sesame Street, as the Cookie Monster learned to deal with his cravings – and taught viewers to do the same. “Me want it but me wait,” he sang in the accompanying electropop pastiche. But does delayed gratification always lead to better wellbeing? Signs that this might not be the case were already apparent in the 1990s. Analysing the state of the evidence, Prof David Funder at the University of California, Riverside found that children who scored well on self-control also grew up to be overly reserved and lacked curiosity. He also pointed to research showing that women with very high levels of self-control tend to be at greater risk of depression, for instance. “The correlates of delay of gratification are definitely a mixed bag,” he concluded. Later research suggested that people with the highest levels of self-control may suffer from feelings of regret. They can struggle to appreciate the present moment, and when looking back over their lives, they come to resent the sacrifices they have made. To investigate this possibility, Prof Ran Kivetz of Columbia University and his doctoral student Anat Keinan asked university alumni to reflect on their winter breaks from 40 years previously. The researchers found that the ageing graduates were much more likely to lament having had too much self-control than too little at this key moment in their youth. Their regret over the pleasures that they had missed from being too sensible, such as turning down the chance to travel, was much greater than any guilt over their moments of indulgence – the times they had skipped their studies, spent too much and acted irresponsibly. Interestingly, the researchers found exactly the opposite views among current undergraduates: these students were much more likely to endorse the standard view that self-control was preferable to indulgence. It was only with the perspective of a lifetime that the alumni could recognise how much richer their life might have been if they had practised a little less self-denial. People with low self-control are often said to have a kind of psychological myopia, but Kivetz and Keinan proposed that many suffer the opposite problem – a psychological longsightedness that leaves them so deeply focused on their future goals, they cannot enjoy all the delicious distractions of the present moment. Strategic indulgence Besides ignoring these long-term regrets, historical psychological research might have overstated the short-term consequences of momentary indulgences. According to one prominent theory, any lapse would only encourage more slip-ups, as we find ourselves falling for further temptations. If you are on a diet, for instance, one slice of cake may soon lead to another, until all your good intentions are in tatters. Similarly, once you start watching videos on YouTube, you may find that the whole morning has passed by without you getting any work done. For this reason, indulgences were seen as “failures” that should be avoided. This idea also has religious origins. “This abstinence idea has its roots in Christianity,” says Prof Lile Jia at the National University of Singapore. Yet recent research shows that intermittently giving in to our desires can often be better for our wellbeing, without putting us on a slippery slope to failure. The trick, it seems, is to plan the indulgences in advance. Consider a study of dieters, aptly titled The Benefits of Behaving Badly on Occasion, conducted by Prof Rita Coelho do Vale at the Católica Lisbon School of Business and Economics in Portugal and colleagues at Tilburg University in the Netherlands. All the participants wished to lose weight and aimed to consume an average of 1,500 calories a day. For those in the control condition, there was no opportunity for variation. Those in the “behaving badly” condition, however, were asked to eat just 1,300 calories on six days of the week; they could then enjoy a blowout of 2,700 calories on the seventh. Over the first two weeks, the researchers tracked the participants’ feelings of motivation and their general mood. They then followed up again one month later to find how much progress they had made. As you might expect of people on calorie-restricted diets, the participants in both conditions lost a few kilos. On average, their body mass index dropped from about 25, which is considered overweight, to about 24, which is just within the “normal” category. There were, however, significant differences in their experiences of the diet: the people who had planned those days of indulgence reported more positive feelings and remained more motivated throughout. The participants who simply cut their calories without the treat days, in contrast, seemed to find it much harder work to maintain their self-control and stick to the diet. That could be crucial for a dieter’s long-term success. Jia has noted similar phenomena in his research comparing the habits of students with high and low grade point averages (GPAs) at US universities. He was interested in the ways that they responded to big-time collegiate sports games – American football, basketball and baseball. These are an important part of student life in the US, but also a huge distraction from their studies. If successful self-control simply involves avoiding short-term pleasures in the pursuit of long-term goals, then you would expect the high-GPA students to have shunned the matches in the run-up to their exams. To find out if this was the case, Jia and a colleague at Indiana University Bloomington asked 409 students to take an online questionnaire a week before a home basketball game against a long-standing rival team. They reported their general attitudes to basketball, and then gave an hour-by-hour plan for their studies on the day before the game, the day itself and the day afterwards. Overall, the low- and high-GPA students hoped to devote roughly the same amount of time to studying over those three days; the big difference lay in the way they distributed those studies. The more successful students planned to take much more time off on the day of the game, but compensated for that with a few extra hours on the days either side of the match. The low-GPA students, in contrast, planned to skip the game entirely. Crucially, a follow-up study confirmed that the more academically successful students were much more likely to have actively participated in watching the collegiate games and celebrating afterwards – and this brought significant pleasure. “They were enjoying the activities more,” says Jia. That would have then put them in a better psychological state to continue their studies the next day. Jia’s latest research suggests that the advantages of “strategic indulgence” may come from an increased sense of autonomy – a finding that may be useful for anyone hoping to avoid procrastination at work. Avoiding guilt There are many ways that we could incorporate this new view of self-control by including a few strategic indulgences into our own lives. We can set up pleasant diversions in a long working day or schedule regular treats during our health kicks. If we’re saving money, we may set a date each month to enjoy the odd luxury as a reward for our frugal living. Just as importantly, this research should teach us to look a little more kindly upon those unplanned indulgences that may accidentally take us a step away from our long-term goals. You may think that guilt and self-criticism will help you to learn from your mistakes, but recent psychological literature shows that they are often counterproductive. By increasing our stress levels, and reducing our sense of self-efficacy, these emotions can impair our motivation. You would do much better to treat yourself with a little self-compassion, relishing the pleasure before looking for practical means to get back on track. As the study of university alumni showed, balance is the key – we should aim to weigh up the needs of our present or future selves to ensure that we are serving the health and happiness of each. And contrary to the self-help puritans, a little bit of hedonism is sometimes exactly what you need. David Robson is the author of The Expectation Effect: How Your Mindset Can Transform Your Life (Canongate, £10.99). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply