How did the pollsters get the British elections so very wrong?

A voter arrives at a polling station in the Southwark district of London during the May elections. (Jason Alden/Bloomberg News)
The past 12 months have not been kind to political pollsters. Opinion polls have been well wide of the mark in the U.S. midterms, the Israeli general election, the Greek and Scottish referendums, and the British general election. The U.K. election was perhaps the most embarrassing of all. Through the heavily polled election campaign, surveys showed a tie between the center-right Conservative Party and the center-left Labor Party, indicating that neither would have the seats to form a government on its own. It came as a considerable surprise when the Conservatives won the election by 6.3 percentage points and an outright majority of seats.
In a new working paper, we examine several theories on why the British polls went so wrong. Two of the most discussed theories focus on survey respondent misreporting – about whom they are going to vote for and whether they are going to vote at all. We examine the evidence for and against both of these in this post.

[Read initial assessments of what went wrong with British election polls at The Monkey Cage from one of the authors of electionforecast.co.uk.]

We use data from the British Election Study (BES) Internet Panel survey, the post-election wave of which has just been released. The BES Internet Panel is well suited to studying polling errors, given that the survey is conducted using methodology similar to that used by pollsters and provides an open data set so that analyses can be freely checked by anyone else. It also shares many questions with forthcoming BES face-to-face probability sample, which provides a gold standard to compare the data collected using polling methods to.
Shy Tories
British opinion polls have a long history of overestimating Labor support. One theory is that shy “Tories” (as supporters of the Conservatives are known) tell pollsters that they will vote Labor (or that they don’t know whom they will vote for) but cast their ballot for the Conservatives on election day.
Without looking over the shoulder of survey respondents as they cast their votes, this theory is difficult to assess directly. We do, however, have indirect evidence against the Shy Tories theory. We examine that theory by considering where there is likely to be social pressure on Conservative voters — in short, are the Conservative voters in our survey in the same places as the Conservative voters in the election.
It seems unlikely that Tories would need to be shy in the heavily Conservative-supporting rural towns in the south of England, but it is more plausible that they would be shy in traditional Labor heartlands such as the former mining towns in the north of England. In fact, the graph below shows the opposite pattern. The deviation between the proportion of BES respondents saying they voted Conservative and the actual proportion of voters who did is highest in strongly Conservative areas where we would expect the least social pressure against voting Conservative.

We also find no evidence for another aspect of the Shy Tories theory. Several pollsters have suggested that placing the vote-intention question later in a survey, after questions about how much they like the party leaders and how they rate the parties on different issues, makes respondents more willing to admit that they plan to vote Conservative. In the first three waves of the BES, we randomized the placement of the vote-intention question to be either at the start of the survey or at the end. We find that the question placement makes no difference to the proportion of respondents intending to vote Conservative.
Taking these findings together, we doubt that Shy Tories were a major contributor to the polling miss.
Differential turnout
An alternative theory is that supporters of the Conservatives and Labor say they are going to vote in equal measure, and report having done so, but actually turn out at different rates. Based just on self-reported turnout, there is not much evidence for this: Respondents who said they would vote Labor or Conservative in the pre-election wave both report 96 percent turnout afterward. This compares with 66 percent turnout in Britain as a whole. Although this partially reflects the fact that poll respondents tend to be more politically interested than the general population, we also have considerable evidence that respondents overstate their turnout.
In some parts of the U.K. (but not all), local elections were held at the same time as the general election: 20 percent of our respondents in areas without local elections in 2015 claim to have voted in them. Similarly, 3 percent to 6 percent of respondents in the campaign wave claim to have voted by post before the postal ballots were actually issued, and 46 percent of respondents whom we could not verify as registered to vote in June 2014 claim to have voted in the 2014 European elections. In
all of these cases, the fibbers lean significantly more Labor than other respondents.
We look at the impact of overstated turnout more precisely by building a predictive model of turnout based on validated turnout in the 2010 BES face-to-face survey that accounts for stated likelihood of voting before the election, turnout in previous elections and demographic factors. We estimate that our respondents’ turnout is likely to have actually been around 73 percent. Most important, we can look at how vote intention differs among respondents who have different predicted probabilities of voting.
The graph below shows that the Labor lead among unlikely voters grew hugely between 2010 and 2015, suggesting that differential turnout is an important factor in explaining the polling miss. Considerably fewer of those saying they were going to vote Labor are likely to have actually turned out to vote. Re-weighting our respondents according to their predicted probabilities of voting explains about 25 percent of the gap in the Conservative lead between the pre-campaign wave of our survey and the actual election results. This is mainly due to low-probability voters switching to Labor and away from the Conservatives rather than a change in the enthusiasm of existing Labor supporters.

The U.K. is only one case, and the reasons for the British polling miss do not necessarily explain the inaccuracies of polls in other countries. However, turnout inequality is not just a British phenomenon. It would not surprise us if differential support among low-turnout-probability voters also played a role in other polling misses.
Jon Mellon is  a postdoctoral research fellow at Nuffield College at the University of Oxford. Chris Prosser is a political scientist based at the University of Manchester, where he is a research associate on the British Election Study, and is finishing up his D.Phil. in politics at the University of Oxford.