Six reasons why it's a mistake to compare countries' coronavirus figures

15 April 2020

8:16 PM

15 April 2020

8:16 PM

I was inspired to write this piece by a chart made by the historian James Barr. It showed, in the style of the Financial Times’ graphs of coronavirus cases and fatalities in different countries, a comparison of ‘Apples, Pears and Rhubarb’. What was meant as satire perfectly summed up why it is so unwise to compare coronavirus outcomes in different countries.

Now with new **data**

— James Barr (@James_Barr) April 13, 2020

We are now facing a catastrophic global crisis where everyone is at risk. This is a new disease attacking different countries, with different cultures, different vulnerabilities and different political systems. Yet many people cannot help but track and compare different countries’ death-tolls and cases like it is a league table.

When clinical scientists study medical interventions the primary outcomes we measure are deaths from all causes and the quality of life of the patient. All other outcome measures are surrogate. For example, even if a new treatment saves someone’s life, it may also cause one extra death from the toxicity of the treatment. Or just because a treatment shrinks the shadow on a chest X-ray, it may not mean a life will be saved from lung cancer. I have learnt these lessons the hard way. As well as my role as a professor of surgery, I established and directed the first clinical trials centre for cancer in the UK in 1981 and ran the department for nearly 30 years.

Comparing outcomes in this pandemic between different countries is like comparing apples, pears and rhubarb. There are so many confounding variables (or biases) it’s hard to know where to start, but here are six variables that illustrate the problem with international comparisons of coronavirus deaths and cases:

1. Population density

The population density of New Zealand is 18/square Km whilst the population density of England is 430/ square Km. This will obviously affect the spread of the disease.

2. National demographics including incidence of co-morbidity.

Some minority groups in rich countries will have more co-morbidities such as diabetes, hypertension and obesity. While in India, Pakistan, Bangladesh and Sub-Saharan Africa, malnutrition and infectious diseases are rife.

3. A single or multiple epicentres for the start of the epidemic

South Korea seems to be an exemplar. Yet the country mainly had a single cluster of the disease in a church with a full register of members. Health authorities also had warrantless access to the credit card and phone data of its citizens, including location data. So what we might regard as breaches of civil rights are part of the country’s success story.

4. Recording of incidence of infection 

The more tests for the virus that are conducted, the higher the incidence of the virus appears. Yet some of the apparent differences in the number of deaths worldwide could be related to the type of antigen test used in each country. There are many tests available but there has been no transparency in reporting their sensitivity and specificity. In other words, the tests with the lowest sensitivity (leading to more false negatives) would reduce the number of deaths (suggesting a better outcome for each country), while the more accurate tests would increase the number of apparent deaths caused by coronavirus.


5. Recording cause specific or all cause deaths 

The way deaths are recorded greatly varies between countries. Many deaths, for example, will be in care homes or in people’s homes in quarantine a long way from hospital, and so won’t be recorded. Even if we only measured deaths in the ICU in hospitals to compare different countries, this would be insufficient to accurately quantify the death rates from the coronavirus because many deaths outside this will be wrongly attributed to co-morbidities that make the patient vulnerable to the virus. The Office for National Statistics (ONS) is considered one of the best in the world, so in this respect we are victims of our own success. This view is vindicated by reports today that show an alarming increase in deaths from ‘other causes’ that match the peak in deaths from the coronavirus in the last few weeks.

6. Deaths from the collateral fall-out of the draconian measures taken so far 

As the chief medical officer (CMO) keeps warning, there will be collateral deaths as a result of the measures used to contain the virus. For example, deaths from cancer might be a consequence of redistribution of NHS resources, and deaths from suicide might be the consequences of social isolation. Yet we shouldn’t forget collateral non-deaths, because of the decrease in road traffic accidents, for example. A paradox of this crisis is that our A&E departments are almost empty.

It is necessary to take the long view as well. If the lockdown ends too soon there might be a second wave of the disease and if it ends too late there might be civil unrest, looting and deaths from violence. A paper in the latest issue of the Lancet has studied and modelled the hazard of a second wave of the epidemic in China and concludes that there is likely to be substantial risk of viral reintroduction if we don’t plan ahead. And while a longer lockdown could reduce the risk of a second wave of infection, on the other hand, it might also lead to greater unemployment and poverty.

We now have a rich and varied amount of data on coronavirus that is global and increasing by the day. As you would expect, in the face of uncertainty, opinions in the scientific community are diverse. Now is not the time for point-scoring and facile comparisons, but for global collaboration.

Ultimately this lockdown period will come to an end, we will develop herd immunity, produce antibody tests that work and vaccines to protect the vulnerable. We will win against this virus and our children will ask, ‘What did you do in the great corona war of 2020?’.

The best answers will come from those currently caring for the sick or who work in our science and mathematical laboratories. But if you are a whinging TV journalist demanding, ‘Something must be done’, or an armchair epidemiologist who has no doubt that ‘rhubarb’ is the answer to our problems, you may have to keep quiet and change the subject.<//>

Got something to add? Join the discussion and comment below.

Show comments