Standing on Warren Hill in the morning mist, watching Britain’s finest thoroughbreds thunder past, you realise what makes Newmarket so special. Racehorses are all around you — there are yards all over town. Every morning, there are hundreds of horses out training. And these aren’t any old horses. They’re some of the fastest racehorses in the world.

Horses have cantered across this windswept heath ever since James I came here to hunt, and 400 years later racing remains Newmarket’s lifeblood. The town grew up around racing and its layout reflects its sporting origins — Tattersalls, the world’s oldest bloodstock auctioneers, is just behind the high street. The surrounding fields are crisscrossed by 50 miles of gallops.

The hub of the old town is the Jockey Club Rooms, housed in a grand old Georgian building on the high street. The Jockey Club thrashed out the rules of racing here, back in the 1750s, and oversaw the sport for several centuries thereafter. The gentry used to sit in here and fritter away their fortunes, betting on the horses that raced each other down the high street.  As racing became more organised, the action moved to the Rowley Mile, named after Charles II (aka Old Rowley), who loved his racing, and was a pretty good jockey too.

The Rowley Mile is still going strong — it remains one of Britain’s most famous racecourses — but there’s loads of other stuff to do in Newmarket. You can visit the National Stud, and learn about the delicate and costly business of siring future Derby winners. I went on a guided tour (open to all) and relished the awkward spectacle of a retired jockey struggling to explain the meaning of ‘teasing’ and ‘covering’ to some bemused schoolchildren.

James I built a splendid palace in the middle of Newmarket, and a fragment of this mansion now houses the National Heritage Centre for Horseracing & Sporting Art. Locals simply call it Palace House, which is a far better name for it — you don’t need to be a racing aficionado to enjoy it here. The paintings are sublime and the permanent display about the history of horseracing is fascinating. In the paddock out the back you can meet retired racehorses like Purple Moon, who came a close second in the 2007 Melbourne Cup.

You don’t need to be a member to stay at the Jockey Club. Even though it’s a private members’ club (members include the Queen), when they’re not too busy with race meetings or bloodstock sales anyone can book a room. I blagged a bed in the King’s Suite, where George V and Edward VIII used to stay when they came to Newmarket. As you’d expect, this historic apartment is supremely comfy, and I especially enjoyed the four-poster bed and the huge bath.

Breakfast is the biggest treat. I walked along a long corridor, past endless rows of mugshots of Jockey Club members past and present, to a sunlit dining room overlooking a pristine lawn and busy stables beyond. I ate my eggs and bacon beneath a portrait of the Marquess of Rockingham, one of six prime ministers who have been members here. When I left in the early morning (sadly I had boring office chores to attend to), the horses were already hard at it, cantering away up Warren Hill.