Iam standing in my bedroom in my boxers and a T-shirt, while a man I have just met fiddles with my legs. His name is Parthasarathi, and he’s here with his boss Julius, who has come to set me up for a polysomnogram – AKA a sleep study. Once they have glued electrodes all over my legs, chest and head, and stuck a cannula into my nostrils, and clipped a monitor to one of my fingers, and strapped more electronics to my chest and waist, and trained an infra-red camera on my bed, and given the Guardian’s photographer a good laugh, they’ll be leaving me for the night. Then all this kit will track how long and deeply I am sleeping, how much I am snoring, how twitchy my legs are, how often I get out of bed, whether I talk, walk or … I don’t know, juggle in my sleep, what’s happening to my blood oxygen levels, what my heart’s doing and, crucially, how well I’m breathing. The answer to that last one turns out to be: not very well. I later learn that I stopped breathing for at least 10 seconds that night – not once, not twice, but 60 times. That’s an average of almost 10 times an hour. What the hell? I think. At the same time, I’m comforted. Maybe I’m getting closer to fixing my insomnia. It’s been a problem for at least half my life, and now I’ve decided to live to 100, I don’t want it screwing up my final 40 years. It increases the risk of heart disease, stroke, diabetes and depression, as well as accidents like crashing your car or walking in front of a bus. How bad is mine? Most nights I don’t get much more than five hours’ sleep. I wake at least once in the small hours, often twice or more, and if I wake up any time after 4am there’s a good chance I won’t fall asleep again. I think you’d call me a high-functioning insomniac. I don’t feel terrible most days – just a little tired. It hasn’t stopped me holding down a good job, getting married, having a family. And the fatigue is mainly mental. I can usually manage a run, or an exercise class, or some yoga, as well as a full day’s work. A woman recently wrote to the Guardian asking if I had servants. How else could I fit everything in? It’s simple, madam: I rarely get more than 20 winks. But I wish I did. There are moments when I’m drifting off, almost always when I’m lying on my back, when I feel something I can only describe as bliss. I bathe in that warm ooze of wellbeing, a smile in my heart – until my wife, Hannah, pokes me to complain I’m snoring. There are certainly times that euphoria would come in handy. I’ve had two brief episodes of depression and anxiety, both bad enough to need medication. Both times I was sleeping particularly poorly. I don’t think that’s the only reason I became ill – but it felt as if the misery and the insomnia were feeding each other. Maybe I just need to improve my “sleep hygiene”? Don’t you start. This is all you ever hear about if you’re struggling to sleep, from every expert, book, website, app, audiobook or well-meaning stranger. If you’re lucky enough never to have encountered the concept, it simply means doing everything you can to encourage sleep while stopping everything that may undermine it. Entirely off the top of my head, I can tell you that this includes: 1) making sure your bedroom is dark and quiet; 2) getting plenty of daylight first thing, maybe with a long walk; 3) getting plenty of exercise generally, though not too late in the day; 4) avoiding blue light from screens in the evening; 5) not sleeping next to your phone; 6) turning down the lights as you approach bedtime; 7) avoiding social media and other rage-inducing stimuli late at night; 8) going to bed and rising at about the same time, even when you really, really fancy a lie-in; 9) keeping the lights off if you must get up to use the toilet; 10) keeping caffeine to a minimum, avoiding it completely in the evening; 11) ditto booze; 12) leaving at least a few hours between your last meal and bedtime, so you’re not still working hard to digest it; 13) not eating too much for dinner, for the same reason; 14) using your bed only for sex and sleep; 15) getting out of bed if you’ve been awake for more than 20-30 minutes, then moving to another room, where you should do something soporific; 16) trying not to check the time when you’re in bed; and 17) clearing your mind before bed by making a to-do or to-worry-about list. I could go on. You know what? Following these rules almost certainly will help you sleep better. The same goes for meditation/mindfulness, breathing exercises, yoga and tai chi, all of which I have tried and enjoyed, and all of which have helped a little. They all relax you (in the jargon, “engage your parasympathetic nervous system”) and some you can practise while lying unhappily in bed. Experts quite rightly try to give sufferers the confidence they will one day overcome their problems, not least because worrying only makes it harder to sleep. But the truth is, none of this is guaranteed to get you all the way. Not even the treatment often described as the gold standard: sleep restriction, which is usually combined with most of those rules in a programme of cognitive behavioural therapy for insomnia, or CBTi. The aim is to create a strong association between your bed and sleep (as opposed to reading, working, watching TV, etc – and especially tossing and turning). I recently endured almost six months of this – during two of which I had weekly phone calls with my local NHS CBTi service. Approaches vary a little, but the basic principle of sleep restriction is to work out how much shut-eye you are getting in your horrible, underslept state (five hours, for instance). Add another 30 minutes and then allow yourself only that long in bed for night after night until exhaustion has bludgeoned you into sleeping solidly for at least a week, at which point you get an extra 15 or 20 minutes. Once you’ve proved you can make the most of this, you get another 15 or 20 minutes, and so on until you can’t improve any more. Well done, you! Even before you start all this, it’s clear it’s going to be an enormous pain in the balls. You’re warned that you will be even more tired than usual, and more irritable, and find it harder to focus, and that you should tell your family, friends and workmates so they can make allowances. All of that is true; none of it quite captures the grim tedium of staying up past midnight for seven, 14, 21, 28 nights in a row, exhaustedly observing those loathsome rules. Imagine sitting droopy-eyed in a darkened room, long after everyone else has gone to bed, listening to an audiobook that won’t overstimulate you, and trying your damnedest to stay awake because – let’s call this rule number 18 – it’s not a great idea to nap, and if you do, it shouldn’t be for long, and definitely not in the evening. It’s not quite hell, but it’s definitely purgatory. What you will learn en route is that it is remarkably difficult to be sure how much you are sleeping. We routinely wake up in the night and forget about it by the morning, or even think we have been awake when we have actually been sleeping. This is known in extreme cases as paradoxical insomnia. And remember rule number 16: you must not check the time during the night. As for wearables such as the Apple Watch, the Oura smart ring or the Whoop tracker band (all of which aim to chart not just your hours asleep, but the amount you spent in light sleep, deep sleep and REM sleep), I’ve tried them all and I wouldn’t rate them any more than seven out of 10. They are particularly erratic when it comes to distinguishing between sleep and lying really still. Like when you’re awake and trying not to be. Does sleep restriction work? I’m sure it does for most people. I began to sleep a little longer and with a little less disruption – but I still wasn’t sleeping well. I began to wonder if I ever would. Have I tried pills? Of course, and I’d probably have used them more if I could get the good stuff. Lormetazepam, which I was prescribed when living in France, is rarely dispensed in Britain, partly because of fears of addiction and partly because of other possible side-effects. It worked well for me, and I could genuinely take it or leave it, but the problem with that argument is that the more you repeat it to your doctor, the less convincing it sounds. Amitriptyline, which is readily available, left me muzzy; I binned it completely after one horrible morning of irritable, spaced-out jitters. I have just been prescribed zopiclone, which a friend tells me is wonderful. But this is another drug that GPs are wary of, and I’ve only got seven pills, so I’m keeping them for emergencies. And in any case, pills rarely give you proper, restful sleep. Sometimes a few hours of unconsciousness is all you need to stop spiralling, but it’s not a long-term solution. As for the over-the-counter, non-prescription sleep aids, I have spent a fortune on Nytol, valerian, magnesium, 5HTP, lavender, and black cherry gummies. Nytol One-a-Night (the hardcore version with antihistamines) did some good; the rest were as useless as they were expensive, though the gummies were tasty. And what about the holy grail for self-medicating insomniacs: the hormone melatonin, widely recommended for jet lag and other sleep problems? It’s available over the counter in the US and many other parts of the world, but only on prescription in the UK. I tried it for months, in every dose from 1.9mg to 10mg, in quick-acting and timed-release forms. When it had any effect, which wasn’t always, I reckon it made me sleep a little more soundly and just a little longer. Helpful, maybe. Magic bullet? Not for me. A few weeks ago, I spoke to a doctor who had done a lot of work with sleep. He said I shouldn’t write off the progress I had made with the hygiene and the CBTi and the sleep restriction: these things take time, and was I familiar with the Japanese concept of kaizen, or continuous improvement? You create something, then you add a bell to make it a little better, then a whistle, then another bell … That was a helpful way to think about my achievements, such as they were. Then he told me about a new type of sleeping pill called an orexin receptor antagonist. “How do you spell that?” I asked. Anyway, when I arranged to spend a night decked out like a Christmas tree, it wasn’t because I thought a polysomnogram would help me. Dr Sundeep Chohan’s kaizen seemed like my best – probably only – hope of success. But after being so frustrated by my attempts to measure my sleep, I was interested in the proper way to do it – the test that doctors rely on. Enter a company called Independent Physiological Diagnostics (IPD), which brings polysomnography to your home rather than making you check into a sleep clinic. In between fitting me with all those electrodes and cannulas and so on, Julius Patrick, a clinical physiologist as well as IPD’s managing director, checks my throat and asks me questions designed to help with the analysis. Do I ever wake up gasping for air (no, thank God, and don’t you think I’d have mentioned it?), do I have headaches in the morning (nope), do I get brain fog (sorry, what was the question?), how often do I pee in the night (too often), do I ever have nightmares or night terrors (again, no) and do I dream? Surprisingly rarely, I tell him. Oh, everyone does, he says, but not everyone remembers it. And you know what? No sooner do I fall asleep than I jerk awake and tell Hannah I’ve been trying to catch a spider in a glass. It’s so huge that its legs stick out and it just scuttles away, taking the tumbler with it. Then I’m out cold again. That’s not the real surprise, though. A few days later Dr Oliver Bernath, the neurologist who reviews my test results, announces that I have “moderate obstructive sleep apnoea”. I’d heard about apnoea before, but always associated it with bigger bodies, daytime sleepiness and dramatic symptoms like the aforementioned gasping for air. It turns out to be far more widespread and often less spectacular; according to the Sleep Apnoea Trust, it affects up to 10 million people in the UK, 4 million of them moderately or severely. The vast majority of cases go undetected, so that people like me who have what Patrick describes as “regular arousal from sleep” end up focusing on other causes. That’s not to say you can’t have more than one reason for terrible sleep – hence the term Comisa, or co-morbid insomnia with sleep apnoea. Bernath recommends I get treatment “to reduce the long-term risk of cardiovascular complications”. This “may also improve the subjective experience of sleep quality”. This is, despite the muted language, a gamechanger. I finally have an explanation for my problems – and a way to tackle them. If the worst comes to the worst, I might need a continuous positive airway pressure (Cpap) machine, to feed me air during the night. I admit, my heart sinks at the thought of wearing a mask to sleep, and not just because it feels like a memento mori. As my wife likes to remind me, at 50 she is “considerably” younger than me and I’d rather not give her another excuse to tease me. Then there’s Stevie, our sweet but nervous rescue dog, who usually spends the night under our bed. He already freaks out when one of the kids turns up in a new hat or a novelty wig. He will quite possibly wet himself if he sees me attached to a Cpap machine. But there’s a good chance it won’t come to that. The apnoea only occurs when I’m “supine” – that is, on my back – so I may be able to avoid it just by sleeping on my side. The thought makes me a little sad given how much I enjoy a supine snooze. But it’s clearly going to have to stop. To make that a reality, my options seem to be good old willpower, the sort of V-shaped pillow that’s popular with pregnant women, or a device that will stop me rolling on to my back. This could be one of the traditional anti-snoring hacks – like a tennis ball sewn into the back of a pyjama top – a special backpack, or something electronic that will prod me if I deviate from the correct position. I immediately go online and order the pregnancy pillow, then add a belt that will give me a little electric shock every time I go astray. It’s early days but I’ve been trying a combination of pillow and willpower for the last week, and so far the signs are good. I’m sleeping better than I have for ages – on my side, obviously – and waking up more refreshed. If I start backsliding, I’ll move on to the belt, and then the Cpap machine. My ego, Stevie and Hannah will all just have to suck it up. Sorry, Stevie! Sorry, Hannah! After decades slugging it out with insomnia, I think I’ve got it on the ropes. I’m actually looking forward to the next 14,000 nights.