I celebrated Eid in a sandy bay in Sri Lanka, watching from the warm, shallow sea as gaggles of local Muslims in holiday mood sauntered past to congregate at the public end of the beach about half a mile away. Since they looked so much more colourful, picturesque and exotic than the tourists in the security-guarded enclave where I was, I thought I’d wander down to take a few snaps.

Having just finished Ramadan, they were all very excited — the young men especially. Suddenly, as if from nowhere, a group of dark-skinned boys with wispy beards, bare-chested but in long trousers, had surrounded me. ‘Selfie!’ one of them said — the new universal word. So we put our arms round one another’s shoulders and smiled for the phone cameras. Sweet kids, but for just a second they had given me quite a turn.

As I headed deeper into their territory, this discomfiture grew. Most of the women wore traditional local dresses — bright fabric with lots of red and orange and pink; no headscarves — but here and there you could see the beginnings of a trend towards more austere, Middle Eastern-style dress. Girls in veils sat gazing modestly into the sand. An intense youth dressed all in white like a cleric strode past with a superior air as if he had higher things on his mind than beach frivolity.

Once I’d retreated back to the safety of my hotel, I was disappointed to notice how relieved I felt. This is very unlike me. In the past, I’ve always been one of those travellers on a perpetual quest for the ‘real’ wherever: forever shunning my sanitised, poncy tourist zones in search of ‘the place where the locals go to eat’ and other bracingly authentic experiences.

On occasion this has got me into quite a bit of trouble. The time, for example, in Essaouira when I persuaded the Fawn that it would be culturally enriching for us to drink tea made from dried poppy-heads in the grotty flat of a souk spice trader — with the result that we spent the whole of the next day in a low-grade opium stupor, vomiting. Or the one where we travelled up to Dahab to smoke Bedouin weed with the man who had turned Jamie Blandford on to heroin at Harrow: the journey home across the Sinai was only about an hour, but in my head it felt more epic, dramatic and gruelling than Xenophon’s Anabasis.

What my most memorable travel experiences have in common, looking back, is that more often than not they happened in the Muslim world. Partly this has to do with geography: the Middle East and the Mediterranean being the cradles of civilisation, it’s inevitable that Islamic nations should own so many of the world’s must-see treasures, from Ephesus and Abu Simbel to Petra and Leptis Magna.

Partly too, though, it’s down to cultural inclination. From Sir Richard Burton to Lawrence of Arabia, the Prince of Wales, the Fry’s Turkish delight advert and — traditionally — almost the entire Foreign Office, we Brits have long had a bit of a pash for the world of hookahs, flowing robes, dunes, falconry, scimitars, camels, sheep’s eyeballs and so on.

I’m certainly no exception. I’ve loved my travels in the Muslim world — the hospitality, the food (well, some of it), that slightly guilty thrill you get from being in a realm where possessing a penis is not a cause for shame — and even in the edgiest of situations I don’t think I’ve ever felt truly threatened. Call me naive, but I’ve always been buoyed by my faith that Muslims, on balance, will do the right thing by strangers in their lands. This trust has so far always been repaid: the man who accosted me by the Pyramids and lured me through the maze of back streets really did just want to make me tea and show me his home; the Jordanian police really were true to their word when they vowed to catch by sundown the silly donkey guide who’d tried to sexually assault my then girlfriend.

That was what I found so depressing by my response to those women in headscarves and the young men dressed like clerics on the Sri Lankan beach. This is the first time in my travels, I think, when I’ve seen Muslims as a cause for mild trepidation. I wish it weren’t so, first because it’s so unfair and disproportionate (only a small minority of Muslims anywhere in the world and probably none in Sri Lanka would ever consider doing something like the Tunisian beach atrocity), and secondly because of what it means for the coming generations, both theirs and ours.

On our kids’ side of the equation, it means no more: Lamu, Kenya, where my brother got stung by a scorpion; Sudan, where those army colonels barbecued a sheep they’d slaughtered in front of us before sharing some of the weed they’d confiscated on drugs patrol; Egypt, where I scaled the Great Pyramid before dawn so as to be on top for sunrise when I could also read the graffiti carved into the stone by Napoleon’s troops; maybe not even Morocco, where apparently Isis are busily trying to recruit ‘lone wolves’ for the next Tunisia-style hit-job.

And on these countries’ side, of course, it means fewer tourist pounds and dollars, less bridge-building contact with the people they’re increasingly being taught to see as despicable kufar, and more of the poverty which acts as a seed bed for increased radicalism.

It’s so sad for all of us and I see no end in sight. No more hippie trail. No more exotic Gap Yahs. In this supposed age of globalisation, the world you can safely explore — or, in cases like the Bamiyan Buddhas or doomed Palmyra, that you can ever see again — has shrunk not grown. Shame.