Fifty years have passed since the death of my father, Evelyn Waugh. His remains, together with those of his wife Laura and daughter Margaret, are buried within a ha-ha which is now collapsing into the churchyard of St Peter and Paul, Combe Florey. My nephew, Alexander, and I hope that these graves could be incorporated in the churchyard as only a dilapidated wall separates them. But our efforts have been frustrated by bureaucratic obtuseness.

I wonder if the creakiness of the bureaucratic process has been created by the undeserved popular perception of my father as a monster. The portrait is based on his own diaries and my late brother Auberon’s wonderfully comic autobiography, Will This Do? The latter presents a quixotic version of the truth, containing among many other anecdotes a story about Evelyn devouring the wartime banana ration intended for his children. This, it’s true, had a reprise in my lifetime — transformed into caviar. One Christmas an American heiress, Mrs Cutting, had decided to adopt our needy English family and had sent a Christmas hamper which included a small pot of caviar. This my father consumed solo in front of his six beady-eyed children. Maybe it was a little greedy, but what fortitude! Most fathers would hide it to share with a significant other when the crowds had dispersed.

I want to mount his defence — but first I should perhaps review the case against. My first memory of my father was blissfully blowing a whistle as loud as I could outside his library window. I had just learnt to blow a whistle and was demonstrating my prowess. I was four years old, my father 51 and no doubt in the throes of writing Officers and Gentlemen, the second book of the Sword of Honour trilogy. Suddenly there he was, incarnate, behind me, fierce but silent and firm, and the whistle was gone. My father had carelessly chucked it under a black camel that lay beyond the green baize door. I was not impressed. Being only four years old I had not acquired the gentry status that would permit me to go through the door and retrieve it. For weeks, I would gaze tantalised from the servants’ side, not daring to approach the whistle nestled in the hind legs of the kneeling camel. In recognition of this early trauma, the camel was bequeathed to me and now my grandchildren ride it with scant respect for its awesome past.

The second item in the case against is not so much a memory as a photograph: myself sitting on top of a stone pillar glaring at my father, who gazes, beatific and adoring, at his youngest child. I have often wondered about that photograph. It now occurs to me that I was only obeying a strict family rule that none of us should ever smile in front of a camera. This was a rule made by my father but invariably broken by him in family photographs. And so our family portraits display a seraphic Evelyn surrounded by a very cross family. It was only in the last family photographic session of his life that those who had reached the age of 20 had started to sneak shifty smiles at the camera. But, in this last photograph, my brother James and I still glared. I believe that James, true unto death to his father’s prohibition, has never ceased to glower at a camera.

My experience of my father was entirely different from that of Auberon, who had to overcome our father’s reputation to find his own literary voice. This he did brilliantly by transferring our father’s talent for surrealist commentary and sharp wit from novels to journalism, an area in which Evelyn did not excel.

My primary impression of my father was of a gentle melancholic man whose chief pleasure lay in parodying his condition. He would wander round the house when too many children were present chanting…

Oh the hell of it
Oh the smell of it
Oh the hell of the family life.

And we would smile indulgently. When his friends died he would cheer feebly, because he felt doomed and he had outlasted them in the race of life. When Ian Fleming snuffed it, he even acted out his death-rattle during Christmas charades. How we laughed.

He never hit his children, the boys were not expected to call him ‘sir’ (a custom among my schoolfriends of that time which I found very incongruous), and he was tolerant of fibs as long as they were fantastical enough and not malicious. Once, inspired by reading Nada the Lily, I took the role of Umslopogaas, using my father’s best cork-topped walking stick as a knobkerrie to vanquish the furniture in the redundant servants’ parlour. The victory, if total, was brief. The battered walking stick was found and all fingers pointed at me.

An impromptu court was set up in a tatty glass shed at the back of the house, which my father had christened the Crystal Palace. He took the role of judge and was provided with a chair. I admitted using the walking stick but hotly denied doing anything violent with it. I made no mention of Umslopogaas and maintained that I had used it to act the French courtier. When asked what that might involve, I minced around in the grime of the Crystal Palace striking what I imagined to be courtier-like poses with the stick. I was absolved, I think on the grounds of chutzpah, and the court was adjourned.

Certainly I was in awe of my father. This was less from fear than from a desire not to appear foolish in front of him. But in my teenage years I felt protective of him. He was fragile, like a beautiful piece of china. Being alone at home, I used to sigh frequently and despondently. Papa could not bear it and threatened to kick me if ever I sighed again. The day came when some of us were standing round the kitchen table and I let out a heavy sigh. ‘Right,’ said my father, ‘I am going to kick you.’ He waddled towards me and I ran away. He waddled on and I could not bear it. So I stopped and let him kick me. It was a gentle kick and not long after he was dead.

He was buried in the ha-ha that used to lie on his land, which has since been sold. This year Oxford University Press will start the mammoth task of publishing a scholastic edition of all my father’s writings. Equally important — for his family at least — is that we surmount whatever prejudice there may be against him, to see his earthly remains incorporated into the churchyard at Combe Florey, where we can visit them.