In the late summer of 1917, following the first signs of the tuberculosis that would kill him within a decade, Franz Kafka went to stay with his sister in the Bohemian countryside. During this unexpectedly calm period in an otherwise perennially besieged life, he wrote a series of aphorisms. One of them runs: “The true path is along a rope, not a rope suspended way up in the air, but rather only just over the ground. It seems more like a tripwire than a tightrope.” He might have been describing the path to the true Kafka, which writers, biographers and academics have been attempting to chart ever since he died. Even Reiner Stach, author of the definitive Kafka biography, chose to end that nearly 2,000-page work on a note of uncertainty, quoting the Prague writer Johannes Urzidil, who said Kafka’s intimates could theorise about what his work meant, but none could say how he came to write it. Ambiguity, mystery and radical interpretability are inextricable parts of works such as The Trial, The Castle, and The Metamorphosis. Is the cloth salesman Gregor Samsa literally a cockroach, or is his transformation symbolic? The brilliance of the story is to allow both things to be simultaneously true. Kafka’s German is famously plain and clear, yet works to enshroud his outlandish scenarios with a paradoxical mystery. “The limpidity of his style,” Vladimir Nabokov noted in his Cornell lecture on The Metamorphosis, “stresses the dark richness of his fantasy.” Might this limpidity mean the answers to at least some of the riddles he poses can be found in the diaries he kept between 1909 and 1923? They have been available in English since the 1940s, but only in a version edited – or, more accurately, bowdlerised – by Max Brod, who defied Kafka’s wish that he burn his writings and instead shaped them to present their author, falsely, as a religious thinker. A restored edition of the diaries appeared in Germany in 1990, and is now available to English-speaking readers via a translation by Ross Benjamin. Benjamin’s aim is to catch Kafka in the act of writing, and to present the diaries not as a cohesive whole, as Brod’s version does, but as “Schrift, writing as a fluid, ongoing, goalless activity.” To this end we get spelling mistakes, scraps of abandoned stories, entries that break off in mid-sentence, and, due to Kafka’s habit of rotating between notebooks rather than writing in one until it was finished, an achronological experience in which we might read the back half of a story 200 pages before its beginning, or pinball from 1912 to 1914 and back again. Brod’s version smoothed such irregularities away, as well as prudishly cutting anything sexual. The Kafka whose posthumous reputation Brod did so much to control, until death loosened his grip in 1968, was not a brothel visitor, nor someone who would describe a male Swedish tourist’s legs as so taut “that one could really only run one’s tongue along them”. More important, in terms of changing the uniquely intimate experience the diaries offer, was Brod’s decision to excise the fiction. One of the book’s greatest pleasures is to read a dull list about who Kafka wrote letters to the day before, then turn the page and discover the first draft of The Judgment, the story that marked a revolution in his work. With it, Reiner Stach writes, “Suddenly … the Kafka cosmos was at hand.” A hopeless figure prey to random punishment or hostile authority, a horror established on the borders of comedy, a plot with one foot in reality and the other in dreams; the seams Kafka would mine for the next 11 years are all here, and we feel, and share, his excitement in the next entry: “This story ‘The Judgment’ I wrote at one stretch on the night of the 22 to the 23 from 10 o’clock in the evening until 6 o’clock in the morning. My legs had grown so stiff from sitting that I could hardly pull them out from under the desk.” Here Kafka seems both genius and ingenue, and the contradiction brings him all the closer to us This new edition restores the variegated richness – and, at times, the tedium – of the diaries: an account of a trip to the theatre might be followed by a story draft, a gnomic half-sentence, the description of a prostitute, time spent watching a ski-jumping competition, relationship problems, dreams of a writing career in Berlin, a list of mistakes made by Napoleon in the Russian campaign, thoughts on the size of a fellow train traveller’s trouser bulge. The muddled presentation of all these elements, contextualised by thorough notes, gives the sense of Kafka not just as “the representative genius of the modern age”, as Benjamin describes him, but also a youngish man finding his way, hungry for experience and inspiration, venting his frustrations and following his interests. Here Kafka seems both genius and ingenue, and the contradiction brings him closer to us. He is a man often distressed about his writing. “Wrote nothing,” runs the entry for 1 June 1912. “Wrote almost nothing,” follows that the next day. On 7 June, “Awful. Wrote nothing today.” The following month, things haven’t improved: “Have written nothing for so long”; “Wrote nothing”; “Nothing, nothing”; “Useless day”. Such are the complaints of many writers, and, like others before and since, Kafka decides at one point his desk is the problem (“Now I have taken a closer look at my desk and realised that nothing good can be done on it”). But there are entries that reveal more profound dissatisfactions. Here we see the person Edmund Wilson called “the denationalised, discouraged, disaffected, disabled Kafka”, self-critical to the point of paralysis. “So forsaken by myself, by everything,” he writes in March 1912, and in 1914 the extraordinary question and answer, “What do I have in common with Jews? I have scarcely anything in common with myself.” Echoes of this sentiment are found throughout his correspondence with Felice Bauer, the woman to whom he was twice torturously engaged, and to his sister Ottla, to whom he once wrote, “I write not as I speak, I speak not as I think, I think not as I ought to think, and so it goes into the deepest darkness.” This might seem like performative self-pity were it not the case that most of Kafka’s works, from The Metamorphosis to A Hunger Artist to The Burrow, the short story he was writing when he died, repeatedly reflect this sense of profound loneliness and isolation. “I am nothing but literature,” Kafka claimed in an entry written on 21 August 1913. Looking at his stories and novels, his diaries and letters, and even the notes with which he communicated in his last days, when the effects of tuberculosis made speaking too painful, the idea his essential self resided more in his writing than his body doesn’t seem entirely hyperbolic. In this light the diaries, in which fiction, confession, dreams, wry humour, and despair combine in a messy, hypnotic network, feel like the closest thing to a path, so like a tripwire, that leads to the threshold of Kafka’s abiding mystery. Diaries by Franz Kafka is published by Penguin Classics (£24). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply.