Seven hundred pages of memoir is stretching it a bit even for an ex-inhabitant of No. 10 with David Cameron’s need for self-justification. Halfway through For the Record I was tempted to skip a chapter or two, but then I encountered a passage that made the slog worthwhile. Talking about his relationship with the Queen, her 12th prime minister notes two essentials in preparing for the weekly audience. First check the BBC news headlines because she is always formidably well informed. Second get up to speed on what is happening in the horse-racing world. (He used to check with his bloodstock agent friend Tom Goff whether one of her horses had won that week or one of her mares had foaled.) The week after Cameron’s father died, the monarch even enquired of a completely flummoxed Prime Minister whether the late Cameron senior’s horse was running at Windsor that evening. It was.

Ruminating on the pleasures of the regular weekend at Balmoral in September, a candid Cameron confesses:

Every year I was asked whether I would like to fish for salmon, shoot grouse, ride one of the Queen’s Highland ponies or go red-deer-stalking. I love doing all those things but I had to cut down on country sports after becoming Conservative leader in 2005. I had enough problems dealing with the ‘posh’ accusation without being photographed with a gun in one hand and a dead bird in the other.

He indulged in all, except the stalking. Once, in Reykjavik, I rode one of Iceland’s special breed of ponies some years after any previous experience in the saddle. So I could sympathise with Cameron’s comment on the particularly broad backs of the Queen’s hard-working ponies: ‘After two hours exploring the hills and glens around Balmoral I would be walking like John Wayne for a week.’

It was a reminder of someone who may be missing racing even more than I do. With Her Majesty and Prince Philip self-isolating in Windsor Castle, one can imagine that having no newspaper version of the Racing Post to prop against the cornflake packets, and no meaningful progress reports from the royal trainers, is restricting the royal interchanges over the breakfast table, even if racing hasn’t always been a subject that binds them. One day when Prince Philip unusually accompanied the Queen on a visit to Ian Balding’s yard at Kingsclere, her carriage-driving husband commented that the horses looked rather thin. He is not the only one in the family who can be direct. The Queen responded sharply: ‘Did you but know, that is how a fit racehorse is supposed to look.’

Racing is and always has been a passionate pastime for Her Majesty and in what would have been the week of the first Classics, the 1,000 and 2,000 Guineas at Newmarket (she won the latter with Pall Mall in 1958), she will surely be suffering true withdrawal symptoms. On the morning of her coronation in 1953, she told a friend that the best news she had had was a favourable training report from Newmarket on her colt Aureole who went on that week to be second in the Derby. Few have seen her as publicly animated as when she wept tears of joy as her filly Estimate won the Gold Cup at Ascot in 2013, the first such victory for a reigning monarch.

Yet while some dismiss racing with a curled lip as an exclusive sport for the rich, and the Queen spends much of her leisure budget on her horses, I cannot find any suggestion that her favourite pastime has cost her any public regard. The answer may lie partly in a report I once came across from Prime Minister Disraeli to Queen Victoria following a conversation with Bismarck. The German Chancellor declared: ‘So long as your population remain devoted to horseracing you have nothing to fear. Here if a gentleman rides down a street 20 persons will say, “Why has that man a horse when I have none?” In England the more horses a person has the more popular he is. So long as the British are devoted to racing socialism has no chance with you.’ The late Labour foreign secretary Robin Cook was scarcely an ardent royalist but he was happy to enjoy a little royal favour by passing on his Racing Post to her after flying out to a Commonwealth conference.

Royal Ascot and Derby Day are the first dates in the Queen’s diary to be ring-fenced every year by the Buckingham Palace team but sadly this year there will be no cheering the procession of carriages from Windsor Castle to the racecourse. Those who run the royal meeting at the Berkshire course in her name have already announced that even if there is a resumption of horse-racing before the scheduled dates of 16–20 June, it will be racing behind closed doors with no public admission. The sooner, then, that we can see those familiar royal silks of purple and scarlet jacket with gold braid and black velvet cap back in action, the better it will be for us all.