Well, you can’t say he wasn’t warned. Swimming pools, Nancy Astor told her son, Bill, were ‘disgustin’. I don’t trust people in pools.’ If he wanted to swim he should take a dip in the River Thames, which flowed through the grounds. But when his horse won the Oaks, Bill Astor built his long-desired pool, and before long Christine Keeler was emerging naked from the water and Cliveden was branded a den of iniquity.

Scandal, it transpires, is in Cliveden’s DNA. The house was conceived by the second Duke of Buckingham, favourite of Charles II, who wanted a love nest for himself and his mistress, Anna Maria, Countess of Shrewsbury, whose husband he had killed in a duel in 1668. The date of the Duke’s triumph over his rival is recorded in brickwork on a patch of lawn between the main block and the east wing, together with a commemorative rapier. When construction began in 1676, Buckingham’s agent compared the building’s ‘crazy foundations’ to those of the Capitol in Rome. By the time it was completed, the duke had been forced to separate from the adored Anna, and so Cliveden’s first chatelaine was Buckingham’s long-suffering wife, Mary Villiers. Gilt leather lined the walls and the colour scheme was a flamboyant gold, purple and emerald green; but instead of being a pleasure palace Cliveden became, as Natalie Livingstone puts it, ‘the melancholy site of Mary’s fractured marriage’.

Cliveden’s next mistress was Elizabeth Villiers, former ‘whore’ to William of Orange and now the sober Countess of Orkney. Under the Orkneys, the house was refashioned as a place of regimental order; despite Pope’s advice on naturalistic garden design, the earl planted his trees to mirror the arrangement of the armies at the battle of Blenheim.

The Mistresses of Cliveden is composed of five biographical portraits; it is also the story of 350 years of English history and oscillating female power. Beginning with Anna Maria, we follow the fortunes of the women who inhabited Cliveden’s walls: Elizabeth Villiers leased the house to Augusta, Princess of Wales, whose husband, Frederick, died before he could succeed his estranged father, George II; following Augusta’s death in 1772, Anne, Countess of Orkney (daughter of Elizabeth Villiers) returned to Cliveden, which was later sold to the Duke of Sutherland, whose wife Harriet was intimate with Queen Victoria. In 1906, Nancy and Waldorf Astor were given the house as a wedding present. Female power really kicked off in 1919, when Lady Astor became the first woman Member of Parliament.

Rolling as it does through the centuries, The Mistresses of Cliveden reminds me of Virginia Woolf’s spoof biography Orlando, which describes the life of an Elizabethan nobleman who lives, without aging, until 1928, by which point he has become a she. Cliveden is similarly ageless: it was twice destroyed by fire (the current house is a mid-Victorian combination of English Palladian and Roman cinquecento), and each incarnation was finer than before. Meanwhile, the tribute to Buckingham’s raffish masculinity has become as soft-centred as an orange crème: in its present incarnation, Cliveden is a luxury hotel owned by Ian Livingstone, husband, no less, of Natalie.

Natalie Livingstone, the latest mistress of Cliveden, describes this book as a biography of the house as well as its female inhabitants, but this isn’t quite the case. She focuses on the grand historical narrative rather than the life of the building. We hear little about the arrangement of domestic space, the world of the servants, the evolution from alternative Hanoverian court to family home. Apart from the occasional intimate scene, we are given a portrait of Cliveden as it appeared from the outside rather than as it was experienced from within; it is less a character than a backdrop for events which take place either at court or in Parliament.

Buried amongst the bevy of hostesses who used it to stage their power are Cliveden’s two most mysterious inhabitants. Mary O’Brien, the daughter of Anne, Countess of Orkney, was born deaf and dumb. Having grown up at Cliveden, she remained steadfastly attached to the property, refusing to give it up to George III. Equally attached was her own daughter, Mary FitzMaurice. After Cliveden was destroyed by fire in 1795, leaving only an outside wall and one wing standing, Mary FitzMaurice moved into the ruin. She remained amongst the rubble until 1821, when she sold it to the Duke of Sutherland. ‘What a hold the place has on one,’ the Duchess of Sutherland later observed.