James Ivory and Ismail Merchant formed the most successful cinematic partnership since Michael Powell and Eric Pressburger. Between the founding of Merchant Ivory in 1961 and Merchant’s death 44 years later, the company produced 42 films, more than half of which were directed by Ivory himself.

Although its range was wider than is often allowed, the company’s fame rests on its adaptation of late 19th- and early 20th-century novels, among them Henry James’s The Europeans, The Bostonians and The Golden Bowl, E.M. Forster’s Howards End, A Room with a Viewand Maurice, and Jean Rhys’s Quartet. Even their detractors — and there are many — acknowledge the wit, elegance and literary sensibility of Ivory’s direction — qualities which are sadly lacking in these memoirs.

Ivory is now 93 and the book, edited by the distinguished American novelist Peter Cameron, whose The City of Your Final Destination Ivory filmed in 2009, gives the impression of having been dictated. What else can explain the clumsy inconsequentiality of sentences such as: ‘Her speciality was chicken pot pie; it is not one of my favourite dishes by any means, but the crust was light and flaky — the mark of a good chicken pie, or any pie for that matter’?

Childhood, often the most vivid part of a celebrity memoir, is here both sketchy and dull. When Ivory was ten his mother told him that he was adopted, but he evinces no interest in his birth parents — and precious little in his adopted ones. He writes more about his Irish and southern ancestors than about his immediate family. Despite hints of childhood rivalry, reignited when he assumes his dead father’s place at the head of the table, his sister merits barely a handful of mentions.

High school and college are recalled primarily for the erotic frustration (and occasional satisfaction) engendered in him by largely straight fellow students. Ivory is admirably frank about sex. Above all, he has an eye for a multiplicity of penises, which inspire his most sustained descriptive flourishes throughout the book. One is described as ‘of the 16-year-old, end of the garden-hose variety;’ another as ‘shaped like the ones on ancient marble statues illustrated in our copy of Will Durant’s The Life of Greece’; a third as ‘a very shapely American frat-boy hard-on, and to my eye the best of the national norm;’ and a fourth as ‘an uncut, rosy, schoolboy-looking ready cock that seemed to match his high-coloured, fair schoolboy’s face’. That last belonged to Bruce Chatwin, with whom he had an intense, three-year sexual (but, he insists, not romantic) affair.

The suspicion grows that, as a director, Ivory was drawn to the works of James and Forster, those masters of emotional repression, because of his unease in expressing his own feelings. He writes little of his 40-year relationship with Merchant, other than to note his handsomeness, his phenomenal energy and his rages. He gives no indication of his feelings about his partner’s many infidelities, save to say that he secured his position ‘like a powerful French matriarch… who knows her husband has mistresses, but who also knows that she is at the centre of his world’.

He is equally reticent about his professional life. He explains that he took a film course in order to evade the draft, and subsequently made several shorts about art. But, except for outlining Merchant’s brash approach to Ruth Prawer Jhabvala for the rights to her novel The Householder, he gives no indication of how he made the leap into filming features.

He claims not to feel the need to know other directors and, apart from heartfelt tributes to his two heroes, Jean Renoir and Satyajit Ray, he refers in passing only to George Cukor and Federico Fellini. Likewise, he offers only tantalising glimpses of his directing practice and, apart from describing the (very different) difficulties of dealing with Raquel Welch and Vanessa Redgrave, few despatches from the set.

The saving grace of the book lies in its series of pen portraits, evidently written decades ago, the finest being a long letter to Prawer Jhabvala on attending the christening of the adopted son of the novelist Lillian Ross and the New Yorker editor William Shawn, with guests including J.D. Salinger. He also provides an extended 1979 diary entry about a formal ball given by the Earl of Pembroke at Wilton, where he watches Prince Charles ‘mopping his brow and looking desperate’ as he evades various young women, and Lord Rothermere threatening his wife Bubbles that ‘we’ll make mincemeat of you if you cause any trouble’ over his Korean mistress.

The wit and sharp observation here confirms the impression that Ivory is happier writing about other people than about himself. This is not the best qualification for a memoirist.