The Track Record of Pre-Election Polls (in 2 Graphs)

by John Sides on October 29, 2012 · 24 comments

in Campaigns and elections

How trustworthy are this year’s presidential polls?  On Monday, November 5, will they be able to tell us who is likely to win the election?  We’ll know soon enough, but in the meantime the historical record provides some important context.  This record suggests three things:

1) The polls have been fairly accurate.  (Adverbs are always a bit subjective, so see what you think after you read the post.)

2) To the extent that they miss, they do so by over-estimating the frontunner’s vote.

3) The reason they miss is not because of late movement among the undecideds but because of “no-show” voters who tells pollsters that they will vote but then don’t.

To show this, I will again draw on Robert Erikson and Christopher Wlezien’s Timeline of Presidential Elections.  Their data include all live-interviewer presidential election trial heat polls from 1952-2008.

Below is their graph of the Democratic candidate’s poll margin plotted against the Democratic candidate’s margin in the actual two-party vote.

Clearly, the polls are very close to the actual outcome.  Erikson and Wlezien write:

From the figure we can see that the polls at the end of the campaign are a good predictor of the vote—the correlation between the two is a near-perfect 0.98. Still, we see considerable shrinkage of the lead between the final week’s polls and the vote. The largest gap between the final polls and the vote is in 1964, when the polls suggested something approaching a 70–30 spread in the vote percentage (+20 on the vote −50 scale) for Johnson over Goldwater, whereas the actual spread was only 61–39.

In very close elections, the polls are still quite close to the actual outcome—missing by 1-2 points at most.   They slightly underestimated Gore’s share of the vote, for example.  Of course, in a close election, 1-2 points is consequential.  But it’s not reasonable to expect polls to call very close elections right on the nose, and small misses shouldn’t be taken as evidence for this astounding claim from former Secretary of Labor Robert Reich:

Pollsters know nothing. Their predictions are based on nothing. They exist solely to make astrologers and economic forecasters look good.

To the extent that polls “miss,” they tend to overestimate the share that the frontrunner will receive.  A similar graph from Erikson and Wlezien shows this:

This is not because of a pro-incumbent bias in the polls.  It is because the front-running candidate typically does a bit worse on Election Day than in the final polls, and that candidate is typically the incumbent:

This figure shows a clear tendency for the late polls to exaggerate the support for the presidential party candidate; that is, most of the observations are below the line of identity relating the polls and the vote. From the figure it also is clear that the underlying reason appears to be that leads shrink for whichever candidate is in the lead (usually the incumbent party candidate) rather than that voters trend specifically against the incumbent party candidate. When the final polls are close to 50–50, there is no evident inflation of incumbent party support in the polls.

Another observation is, again, that the largest “misses” are typically in the order of a 1-2 points (with the exception of 1964) and rarely suggest a different candidate will win than actually won (this appears to have happened only in 2000).  Ultimately, across these 15 elections, Erikson and Wlezien find no evidence of systematic bias, either for or against the incumbent’s party or one of the major parties.

In short, the polls are pretty good.  Obviously, this judgment is somewhat in the eye of the beholder, but I think they deserve more credit than they often get.  For example, I would not quite agree with Jay Cost’s assessment (here, here) that the polls “screwed the pooch” in 1968, 1976, 1980, 1992, 1996, 2000, and 2004.  Granted, Erikson and Wlezien are simplifying here by looking only at the two-party vote, thereby ignoring third-party candidates (e.g., in 1968, 1980, 1992, and 1996).  But the two-party vote is still a useful diagnostic, particularly in a year like 2012 where there is no salient third-party candidate.

Finally, as I mentioned previously, Erikson and Wlezien investigate why the frontrunner in the polls tends to do slightly worse on Election Day.  This is mainly not about late movement among the undecideds but about “no-shows” who simply fail to vote:

Relatively few voters switch their vote choice late in the campaign. Relatively few undecided voters during the campaign end up voting, and those who do split close to 50–50. Moreover, there is no evidence that partisans “come home” to their party on Election Day (although this phenomenon occurs earlier in the campaign)….[O]ne additional factor…explains why late leads shrink. Many survey respondents tell pollsters they will vote but then do not show up. These eventual no-shows tend to favor the winning candidate when interviewed before the election. Without the preferences of the no-shows in the actual vote count, the winning candidate’s lead in the polls flattens.

Given the state of the national polls right now—a virtual tie—the historical record suggests that they will be very close to the eventual outcome, but could be off by 1 or even 2 points, depending on the vicissitudes of turnout.  Given the historical lack of systematic bias toward either party or the incumbent party in close elections, it’s difficult to predict which “direction” the polls might miss this year, if they miss.  The national polls tend to overestimate the frontrunner’s vote share, but with the national polls tied, there is no clear national frontrunner at the moment.

If this same pattern occurs in the state polls, then the battleground state outcomes should be narrower than the polls reflect—movement which might mitigate Obama’s apparent lead in Ohio an Romney’s apparent lead in Florida, for example.  None of this leads to a dramatically different conclusion than you’ve been reading—the election will be close, turnout matters, etc.  But I don’t think discrepancies between the polls and the outcome, in the past or in 2012, reflect or will reflect a massive failure by the polls—what we might call a”pooch-screwing”?—only the inevitable challenges that arise when using a very useful but imperfect instrument to predict the future.

{ 24 comments }

Joel W. October 29, 2012 at 3:21 pm

My read of the “front-runner” bias in the polls when taken to the state-level is that we should see closer margins in deep blue and deep red states, while having no bias in the swing states. The micro-level motivations, I assume, is to not show because you assume a blowout. I don’t expect an overstated margin in close swing states.

Dave Brown October 29, 2012 at 7:58 pm

That is how I read the data as well. The results peel off the 1:1 line at higher predicted margins of victory (Fig. 5.2) but not in the 0-5% range where the battle ground states are currently polled. If this effect is important, the states to watch out for might be MN or PA, states where Obama is far enough in the lead for Dems to feel (over) confident but close enough to potentially swing the other way.

I wonder to what extent having a close race nationally tends to get everyone more involved, even if their state is not a battle ground.

PJR October 29, 2012 at 3:25 pm

The analysis is limited to polls taken in the final week before an election. That’s pretty useful for judging the estimation accuracy of polls, but it’s a low bar for judging prediction accuracy. There’s a difference. (Reich, btw, said prediction, not estimation.)

John Sides October 29, 2012 at 4:09 pm

PJR: So if the final week’s polls are typically close to the outcome, Reich is right? I don’t follow.

PJR October 29, 2012 at 6:35 pm

Meaning that polls are taken for many weeks and months before an election, and the predictive value of polls can’t be judged by looking only at the ones taken in the final days (or hours or minutes for that matter). Reich’s statement may be inaccurate for polls taken at the final hour, but that does not mean it is incorrect overall. Polls are samples that estimate population parameters; even if the parameters are known with certainty at a given time, that does not necessarily imply predictive power regarding the future values of these parameters. I believe that looking at polls in the final week is a good way to test how well the samples are estimating the population parameters as the election draws near. That probably means they are estimating population parameters well during earlier weeks/months, too. But prediction is a different challenge.

John Sides October 29, 2012 at 7:42 pm

PJR: But we’re having a debate *right now* about whether the polls *right now* will be predictive. That’s why the focus on the last week’s polling is appropriate. Elsewhere in E&W, they examine the predictive value of polls over the course of the election year, and show how their correlation with the eventual outcome grows. And, while it’s true that polls use samples to estimate parameters, there’s nothing inherent in that process that makes them ill-suited for prediction. What E&W show is that the polls do pretty well as predictors.

Doug Hess October 29, 2012 at 8:31 pm

John: How well does the Iowa Election Market work? It has one market for buying shares on the popular vote outcome and one on winner of the election.

Nadia Hassan October 30, 2012 at 11:33 am

Interestingly enough, the same (wonderful!) duo has done work on betting markets, and their research implies that markets add little to polls this far out in the cycle!

PJR October 31, 2012 at 6:48 am

Power has returned. Anyway, I like polling analysis but react to blanket statements like “polls do pretty well as predictors.” A poll that estimates the (aggregate) voting intentions of a population should be a good predictor of voting behavior to the extent that these intentions do not change prior to voting. I think the data show that polls are pretty good at estimating voting intentions, if we make the highly reasonable assumption that voting intentions become better predictors of voting behavior as an election approaches. Regardless, keep up the good work.

Ben October 29, 2012 at 3:51 pm

I would suggest if we do see any error, it probably will be the result of likely voter screens. The survey of voters conducted by American National Election Studies (http://electionstudies.org/) for registered voter participation rates over the last three presidential elections (available at http://electionstudies.org/nesguide/2ndtable/t6a_1_2.htm) show that registered Democrats and leaners have turned out at about 98% of the rate of registered Republicans and leaners (average of 89% for Democrats and 91% for Republicans). Given the relatively small difference in recent historic turnout rates for Democrats and Republicans in presidential general elections, it is hard to reconcile the wide disparities seen between results of those polls which report both registered voter results and likely voter results.

scott October 29, 2012 at 4:14 pm

There seems to be vote rigging in presidential elections, including
2004 and 2008, that can be seen by looking at statistics.
Fraudulently, a computer program (possibly at the state’s central
precinct tabulator level) automatically flips a percentage of votes
from one candidate to another but that the percentage of votes flipped
varies with the size of the precinct. The “rules” of the fraudulent
vote flipping algorithm are the following:
Very small precincts don’t have any votes “flipped” unless that
precinct is larger than, for example, 350 voters.
The percentage of votes that are flipped is small (such as .01%) for
small precincts and large (such as 5%) for large precincts with a
*gradual* change in percentage “flipped” dependent on the precinct
voter population.
The reason that the perpetrators don’t flip as many votes in small
districts is because a recount will check a random number of precincts
and a smaller precinct is *more* likely to be audited because there
are more of them. So if fraudulent vote flipping flips 5% of votes
from Democrats to Republicans, a random recount of precincts would
show a smaller error – such as 2%.
There is a smooth gradual change in the percentage of votes flipped
between the small precincts and large precincts and is seen in the
data.
The authors of the paper show that the effect does not happen in data
from some counties presumably because the perpetrators did not have
access to those tabulating computers and it is not seen in democratic
primaries. The authors of the study look at income and poverty rates
which are highly correlated with voter choice but do not correlate
with precinct size. This rules out more Republicans living in large
precincts as being the cause of the anomalous data.
To find the article, do a google on the following words: vote
flipping small large precincts central tabulator
Also, here is a link to the article,

http://www.themoneyparty.org/main/wp-content/uploads/2012/10/2008_2012_ElectionsResultsAnomaliesAndAnalysis_V1.51.pdf

rick October 30, 2012 at 1:56 am

Fascinating! Why is this not bigger news? If this is true, it seems to me it’s the biggest story since Watergate.

g mckay November 1, 2012 at 1:02 pm

Stephen Spoonamore – Man in the Middle – Ohio Vote Rigging sleuth

Watch this very important video about the rigging of the 2004 election. It is the most informative and valid info on this subject that I have found.

scott October 29, 2012 at 4:22 pm

I should add that likely voter models for pollsters unknowingly include the fraud built in to their model because they add points to Republicans to account for past *actual* performance – which had the fraud also.

Nadia Hassan October 29, 2012 at 5:36 pm

When I checked the E & W dataset, the 1996 polls seemed to fare a tad bit worse than 1-2 points.

John Sides October 29, 2012 at 7:40 pm

Nadia: So…4 points?

Nadia Hassan October 29, 2012 at 10:41 pm

The final entry was 57.6% and the values hovered around 57.1-57.9%, so less than four. I’m not particularly interested in splitting hairs over it, though.

Some of these numbers were “wow” to me, personally:
http://www2.psych.purdue.edu/~codelab/PollOdds.Table.html

If I had to bet my savings on it, I sure as heck wouldn’t wager on a 1996 error and I certainly agree that statements like Reich’s are not grounded in good sense or well-reasoned analysis of the evidence.

Chris October 29, 2012 at 6:55 pm

This is good stuff. But aren’t these charts based on popular vote? Saying “the polls” are generally not too far off doesn’t tell us a lot about which polls to trust when they say slightly different things, and I assume it tells us nothing at all about individual states.

Most of the arguments going on right now are about the divergence between the national polls and the swing states, which are not impossible to believe, but seem likely to end up a lot closer to each other when everything’s counted.

John Sides October 29, 2012 at 7:36 pm

Chris: The state polling isn’t rich enough to do the longitudinal analysis that E&W are doing here. So it’s a tradeoff: more elections but only national polls. Agreed that there is discussion of state vs. local, but you’re also seeing blanket statements like Reich’s being thrown around. Important to show those are simply not true.

Nadia Hassan October 30, 2012 at 12:08 am

Some posts by Nate Silver may be of interest to you.

http://fivethirtyeight.blogs.nytimes.com/2012/09/29/poll-averages-have-no-history-of-consistent-partisan-bias/

See also state polling averages usually call it right.

Stephen Reed October 29, 2012 at 10:28 pm

The results analysed by Erikson and Wlezien are less relevant to today’s pollsters because they fail to include Rasmussen – who performs automated land-line polling. Rasmussen is a prolific pollster whose published state polls affect the quantitative poll aggregators, e.g. 538.

Over the past few election cycles, Rasmussen has been shown to quite far from actual results – in the direction of consistent Republican bias.

Robert October 30, 2012 at 10:37 am

I have read that nine of ten persons polled hang up or otherwise refuse to participate. No where in this article is an analysis of which persuasion is more like to hank up on the pollster. I am betting that it is the more affluent voter who doesn’t wish to be bothered, thereby offsetting the higher penetration of cell phone primary users among the traditionally Democratic voters. Basing poll results on one of ten respondents is not a satisfactory way to determine results.

We shall soon know.

Jinchi October 31, 2012 at 10:33 am

“To the extent that polls “miss,” they tend to overestimate the share that the frontrunner will receive.”

According to your graph, that doesn’t seem to be true for close elections. 4 of the 5 fall slightly above the line. My guess is that in a close election, people who call themselves “likely voters” actually get to the polls.

Nadia Hassan October 31, 2012 at 2:16 pm

The authors remarked on this in their book…

Comments on this entry are closed.

Previous post:

Next post: