We used to sneer at the way the Russians were chased out of Afghanistan by a ragtag of mujaheddin armed only with Kalashnikovs and American Stinger rockets. No longer.

The last superpower to be defeated in Afghanistan withdrew in good order, having negotiated the arrangements with Kabul and the mujaheddin. They left behind a competent government, and an Afghan army capable of fighting the mujaheddin to a standstill. That government survived as long as the Russians went on supplying food and ammunition. Then the Russian government, close to meltdown, stopped deliveries. The mujaheddin swept to victory and fell straight into a bloody civil war. Many Afghans were relieved when the Taliban restored order.

Most of us understood that the Americans were bound to react violently to the destruction of the Twin Towers on 11 September 2001. But much of what followed was based on the false belief that Al Qaeda needed Afghanistan as a base. Not so.

Al Qaeda have had plenty of other safe havens from which to operate. The way to deal with them is not by bombing civilians abroad, but by making sure we have effective police, intelligence services, and special forces at home. Of course, that won’t stop all terrorist attacks. But most terrorist incidents in America and Britain since 9/11 have been perpetrated by domestic fanatics. The Americans could have achieved their aims by smashing Al Qaeda, leaving Afghanistan promptly, and warning the Taliban that they’d be back if Al Qaeda returned.

Instead the doctrine of ‘liberal interventionism’ took hold, the idea that it was our duty to bring other people the benefits of freedom and democracy that we ourselves enjoy. But violence isn’t the answer. However sophisticated your technology, aerial bombardment kills civilians. You don’t win people’s hearts and minds by slaughtering their wives and children.

Five years in, it was already clear that we would eventually leave in ignominy, mission unaccomplished. In 2006, Russian veterans asked if we knew what we were doing by sending our men back to Helmand where they had been so roundly defeated a century earlier: assuming we were out for revenge, the Afghans were determined to give us a bloody nose. But it was not only that we were ignorant of our own history. Despite the expertise available, we were either ignorant of the way Afghan politics worked, or thought that money and advice could replace understanding. The Americans have spent two decades, three trillion dollars, and thousands of lives trying to reengineer Afghanistan’s politics. They have failed.

People will say that it would have been unprincipled to pull out. Could we really abandon the Afghan women who were on their way to something better, and all those brave and decent people trying to build a more prosperous and genuinely democratic country? We have indeed betrayed them. But it turned out that we had neither the resources, nor the understanding, nor the will to deliver on our promises.

Now the inevitable blame game has started, starting with our intelligence agencies and our policymaking officials. It is hard to judge from outside: these people keep their mouths shut. But ordinary Afghans were doing what people always do in a civil war. As the Taliban – often their own kinsfolk – took over ever more of the country, they avoided choice where they could, and compromised where they had to. Since our own Civil War we have never had to make such choices. But understanding what was going on did not require a huge leap of imagination.

More culpable are the generals. From the start their over-optimistic predictions sounded eerily like those of American military spokesmen in Saigon as the Vietnam war went from bad to worse. Military insiders say that the generals told the politicians that they could deliver whatever was asked of them, fearful they might otherwise suffer at the next budget round. Some of them prospered: the American General David Petraeus is regarded as a success and a pundit. But no fundamental problems were solved by his ‘achievement’, the military surge in Afghanistan. Biden rightly argued in 2009 that such a surge would compound failure. Now he is roundly attacked by those who still believe in ‘liberal intervention’.

Responsible leaders do not go to war without very good reason. Many of the generation after 1945 had fought themselves. Those who took us into Iraq and Afghanistan and meddled in Libya and Syria had no such experience. They combined ignorance, arrogance and synthetic idealism to a remarkable degree. We naturally prefer to make them the scapegoats. But we live in a democracy. It is us who choose the politicians. Part of the blame sticks to us.

Now we are trying to put the blame on the Afghans themselves: this is a gross injustice. We know from much experience that pouring money into a country which does not have the necessary institutions always produces massive corruption: we should have not been surprised when it also happened in Afghanistan. The Afghan army failed to put up a fight. But all Afghans are good at fighting when they need to. The Taliban believed in what they were doing. Their opponents did not.

The Afghans could have been left to sort themselves out twenty years ago, after we crushed the Taliban first time round. Now we are leaving them to do so in much worse circumstances, while we engage in the self-indulgent luxury of wondering how the prestige and effectiveness of ‘the West’ has been damaged.

Here we shouldn’t exaggerate. The US recovered from Vietnam, a humiliating defeat based on an equally false premise. It will remain the most powerful country in the world, surrounded by allies and clients. The Chinese too want to navigate and trade freely. They will still have to take account of American power. But the days of the hegemon are over. The management of the ‘rules based international order’ will now have to be shared. It’s a painful thought, but no one has yet suggested a better alternative.