Sometimes I say I’m Scottish, a claim often greeted with understandable derision. I was born in England, in Hertfordshire, went to school and university in England and, apart from some spells abroad as a journalist, have always lived and worked in England. I don’t even have much Scottish blood. My mother was English, from the West Country, and three of my four grandparents were English too. I have no trace of a Scottish accent. I don’t even know Scotland very well. I have never had a home there and have never lived there. As far as Alex Salmond is concerned, I might as well be Lithuanian.

And yet, Scotland is very much part of me. Though it may not sound like it, the name Chancellor is Scottish. It may have had its origins in Normandy, but for centuries the Chancellors lived in Lanarkshire near to the little town of Biggar, halfway between Glasgow and Edinburgh. Since the early 15th century at least, they are recorded as having lived in a house called Shieldhill, now a hotel known as ‘Shieldhill Castle’ (which it isn’t) and oddly described on its website as having been ‘established in 1199’.

The last member of my family to live there was my grandfather, Sir John Chancellor, who died in 1952 when I was 12 years old. He had been educated in Scotland, at Blair Lodge at Polmont in Stirlingshire, but afterwards became a soldier in the Royal Engineers, rising to the rank of Lieutenant-Colonel, when he was recruited into the British colonial service, serving as governor of three British colonies before ending his career in the 1930s as High Commissioner in Palestine under the British Mandate, a job he used to describe as the same as that of Pontius Pilate. Later he retired to Shieldhill and stayed there until his death. His widow, my grandmother, stayed on there until 1959 when my father, working in London and strapped for cash, decided to sell it. It became a hotel, and it was there that Nelson Mandela stayed for the Commonwealth conference that was held in Edinburgh in 1997.

Shieldhill was a large, mainly Jacobean house set in lovely country in the upper Clyde valley. I would stay there with my family about twice a year throughout my childhood and grew to love it. In the summer we’d play croquet and shoot rabbits and the odd grouse. In the winter we’d have curling contests on the ice and join the New Year’s Eve revelry in Biggar. I was even made to wear a kilt of the Duff tartan, to which as a Lowland family we had a somewhat dubious claim.

About a mile away from Shieldhill, at the foot of a windswept hill called Quothquan, there is an isolated country graveyard with a ruined chapel and a walled enclosure containing the graves of my Chancellor forebears. Several of these had died alarmingly young of diseases contracted in insalubrious corners of the British empire. The feeling of this graveyard is as British as it is Scottish, although, as far as I know, my father was the first member of the family to have been educated outside Scotland. He, too, was buried at Quothquan in 1989, and is likely to be the last member of the family who will ever lie there.

Most readers of this column will probably know by now the outcome of the independence referendum, but a ‘yes’ vote, if it has happened, would feel like a punishment for my family’s abandonment of its native land, even if this did not feel to us like abandonment when we were all citizens of the United Kingdom. I may not be very Scottish, but still perhaps rather more so than most of those English or other recent immigrants to Scotland who would now carry Scottish passports and pass cockily through border controls. And there are millions of Scots in England who would surely feel an even stronger sense of unjust rejection than I do.

The allure of Scotland has always been strong for anyone with roots there. Even my father, after a life spent living and working in England and abroad, would talk in his retirement about ‘returning’ to live in Scotland, to be once more among his ‘ain folk’ (as the sentimental song puts it). But I don’t think the citizens of a Salmond-led, independent Scotland would feel like one’s ‘ain folk’ any more.