The problem with polling

22 July 2021

8:10 AM

22 July 2021

8:10 AM

If you did an opinion poll about opinion polls, chances are most people would recognise the limitations of market research, offer some unfavourable views of pollsters and deride the uses to which their work is sometimes put.

Yet if you asked politicians and the media whether polls deserve our attention, they would almost unanimously agree. Even after Brexit. Or Trump in 2016. Or the eye-popping poll earlier this month that found that one in five Brits support having a nationwide 10 p.m. curfew permanently in place, regardless of whether or not the pandemic is still raging.

Polls have major shortcomings. Even if pollsters avoid leading questions and interview the perfect cross-section of society, talk is still cheap. It’s a truism of economics that you cannot take at face value what people say. It’s why there’s a wealth of evidence showing that stated and revealed preferences rarely align.

For most goods, for instance, expenditure surveys and national accounts show broadly the same pattern. But less than half the recorded cigarette purchases show up in the Living Cost and Food survey. If we judged people’s desires by their behaviour — as economists do — we would not conclude that pristine health or self-improvement is their only goal. It would be reasonable to assume that many who expressed interest in learning a language or getting fit during lockdown are still sedate monolinguists.

While it can be useful to know which political party people are likely to vote for, new research from the Royal Society lays bare the pitfalls of using polls to show preferences on more complex issues. It calls for caution on how we interpret simple surveys, as there might be consequences to inferring that people either “support” or “oppose” a given measure.

Time and again, snap polls have indicated high levels of public approval for lockdowns. In July last year, contrary to concerns over ‘behavioural fatigue’, one poll showed that half of Brits thought the original measures imposed on 23 March were introduced too late and eased too quickly. Twelve months and two shutdowns later, one in four want nightclubs and casinos closed in perpetuity. They’re presumably the quartile least likely to step foot in either, but the results have enabled the media to create a particular narrative about our society, backed up by simplified numbers.

The trouble is, people use polls — along with the severity of restrictions — as indicators of the threat level. The Royal Society study found that the more people do this, the more they support lockdown. This feedback loop wouldn’t be so problematic were it not for the fact that government only knows what it believes after testing an idea or decision on an audience. Why else would Boris Johnson have U-turned on staying in isolation after being pinged? Had ‘the science’ fundamentally changed? Could those of us in the vanishing minority who still owned the app now disregard the rules?

Public opinion and policymaking have developed a near-symbiotic relationship during this crisis, each influencing and validating the other. The Prime Minister’s failed attempt to circumvent self-isolation isn’t the only example of this. Traditional and social media amplified public outrage over Matt Hancock’s misdemeanours to such a degree that his position became untenable. A YouGov poll published on 14 June 2021 found 71 per cent supported a delay to the roadmap — which was shortly thereafter announced. Though many in government loathe face masks, their messaging changed at the start of the month, around the time that polls were showing a big majority wanted them to remain compulsory.

But could the British public actually be more ambivalent than authoritarian? The Royal Society research found that while the majority of people support most public health measures, they also see significant side effects, and consider many as unacceptable in any cost-benefit analysis. Interestingly, participants use their general assessment of the threat, but not their personal concerns, to guide their support for lockdown. These findings offer hope that, at some point, the scales will tip. Government can only ignore the cost of Covid measures for so long — even if most polls fail to illustrate that people can hold conflicting opinions at any one time. The NHS backlog is the longest on record. Businesses have been clobbered, education disrupted, livelihoods destroyed.

With restrictions still hanging over us and the prospect of more later this year, we’re a long way from regaining the freedom that Reagan famously warned was only a generation from extinction. The former president asserted that it must be ‘fought for, protected, and handed on for them to do the same’. He was right — even if it might mean the public, or the polls, are wrong.

Got something to add? Join the discussion and comment below.

Show comments