This is a guest post by Samuel Best, Associate Professor of Political Science at the University of Connecticut, and Brian Krueger,Professor of Political Science at the University of Rhode Island. The information about voters in this post was drawn from Gallup, CNN, and dimpledchad.info, which is a new website launched by the authors and Jeffrey Ladewig and Clifford Vickrey. Using information derived from U.S. Census surveys and national exit polls, Dimpled Chad displays the behavior of a wide range of specialized demographic groups in federal elections spanning as far back as 1972.
*****
The American public has unprecedented access to the results of public opinion polls. Indeed, if individuals spend any time reading, listening, or watching news, they could hardly escape reports of the latest measurements of the public mind. Almost a century ago, theoretical physicist Werner Heisenberg taught us that we cannot measure something without simultaneously disturbing the object of our measurement. Increasingly, campaigns, media surrogates, and pundits agree that public opinion polls don’t just measure public opinion, these polls also shape preferences and influence behaviors ranging from turning out to vote to donating to campaigns. Accordingly, favorable polls get lauded and unfavorable polls dismissed as flawed or purposively biased.
Since Mitt Romney wrapped up the nomination, Barack Obama has led in most national polls. Predictably, supporters of the Republican ticket questioned the accuracy of the polls. During the fall campaign, these poll doubters were often dismissed as anti-scientific, Republican wishful thinkers, conspiracy theories, or base partisan hacks. Nate Silver became so frustrated at the broadside against the polls that he even harnessed his inner Mitt Romney and bet thousands of dollars that the poll averages would prevail in predicting the election winner. For Silver, only systematic bias across the many survey houses could lead to a Romney victory.
But one venerable polling firm, Gallup, offered the poll doubters hope, by consistently reporting that likely voters supported Romney over Obama, sometimes by wide margins. Karl Rove’s Wall Street Journal opinion piece and Josh Gordon’s National Review article published in November 2012 use Gallup’s results to predict a Romney win. The argument is straightforward. Gallup, unlike every other major, non-partisan polling outfit, anticipates that those who identify as Republicans will comprise a greater portion of the electorate than Democrats. And because about 90 percent of both Democrats and Republicans vote for their party’s nominee, small shifts in the partisan balance of the sample create substantial changes in the overall estimate. If Gallup is correct (and everyone else is wrong) about the partisan composition of the 2012 active electorate, it would represent the type of systematic bias that Nate Silver warns could sink the usually sound predictions from poll aggregation.
Fortunately, and to Gallup’s great credit, a few days ago they reported their final expectations about the partisan make-up of the 2012 active electorate. It should be noted that Gallup does not weight their sample based on a previously identified parameter about the distribution of partisans in the 2012 active electorate. Instead, Gallup uses a likely voter formula that identifies which respondents are likely to vote and which ones are not likely to vote. If more Republicans than Democrats happen to be identified as likely voters, then Gallup will have more Republicans in their likely voter sample. This year, Gallup expects the active electorate to comprise 36 percent Republicans, 35 percent Democrats, and 29 percent Independents (including leaners). A review of eight other major polls revealed that none showed a Republican partisan advantage, with an average Democratic partisan advantage of 4 points.
Now that the exit polls are public, we can assess Gallup’s expectations about a Republican partisan advantage, the lynchpin of Gallup’s lonely prediction that Romney would win the popular vote. The exit polls, while having much higher response rates than conventional polls, and remain our best guess about the composition of the active electorate, are not perfect. Exit polls suffer from the uncertainty and potential bias inherent in any survey. Exit polls use both phone surveys to estimate early voting and a more traditional stratified probability sample of voters leaving their polling place. Each type of sample is combined (totaling more than 20,000 cases) and weighted with actual election results. Because exit polls use a sample of all voters, the results will not flawlessly represent all voters. Even so, because of the high relative response rate, the ability to regularly directly identify voters, and the advantage of weighting based on actual vote returns, the exit polls are unmatched as a tool for describing the voting population.
The graph below shows the 1984 to 2012 exit poll trends in the partisan composition of the active electorate and the Gallup partisan composition projection for 2012. Happily, the exit polls, just like Gallup, report partisanship by excluding independents who lean toward a party. The circles represent Gallup’s projections about the partisan composition of the electorate and the trend line is the historic partisan composition of the active electorate according to the exit polls. Gallup expected the active electorate to contain 35 percent Democrats, 36 percent Republicans and 29 Percent Independents. Gallup’s two partisan projections missed badly. Democrats comprised about 38 percent of the exit polls, which was in line with their multi-decade average and 3 points higher than Gallup’s projection. The partisan composition trend makes clear that 2004, not 2008, was the outlier election year for Democrats. Republicans comprised just 32 percent of the exit polls, about 4 points less than Gallup’s expectation (36 percent). The 32 percent Republican share of the electorate matched 2008, but was down from the peak in 2004 (37 percent) as well as the multi decade average of 35 percent. Again 2004 appears to be the most different case from 2012, not 2008. Because partisans overwhelmingly supported their party’s candidate (over 90 percent for both), Gallup’s expectation that more Republicans would cast ballots than Democrats doomed their overall election prediction. It is worth noting that the multi-poll-average partisan composition advantage of 4 points nailed it (see above). Chalk up another win for poll aggregation.
We know that Gallup did not force their likely voter sample to fit a pre-determined distribution of partisan identification. What then led the Gallup likely voter formula to oversample Republicans? Some clues are found when also considering Gallup’s projections about the proportion of women, young adults, and whites in the active electorate. The graph below shows the trend in the proportion of women among voters from 1984 to 2012 and Gallup’s projection about the proportion of women in the 2012 active electorate. Gallup expected 52 percent of voters to be women, whereas the exit polls suggest 53 percent of 2012 voters were women. Gallup’s expectation about the proportion of women in the active electorate does not appear to be the prime culprit explaining Gallup’s error. Gallup missed by just a point and women were largely split in their vote, giving 55 percent to Obama. Replacing 1 percent of men, who cast 45 percent for Obama, with women would have scarcely moved the overall Gallup estimate.
The graph below shows the trend in the proportion of non-Hispanic whites in the active electorate from 2000 to 2012 and the Gallup projection about the size of the non-Hispanic white voting bloc in 2012. The exit poll race question is only consistent with Gallup over the past four elections. Accordingly, the trend goes back as far as 2000. Gallup estimated that non-Hispanic whites would comprise 78 percent of voters in 2012. This would have been the highest proportion of whites in a presidential active electorate since 2000. Given the secular downward trend in white composition, this Gallup expectation never had a chance. In 2012 the exit polls indicate that 72 percent of voters were white, 2 points less than 2008 and a full 6 point lower than Gallup’s expectations. Because about 80 percent of non-whites cast ballots for Obama compared to 39 percent of non-Hispanic whites, Gallup’s 6 point racial composition gap explains much of their overall error. Simply put, Gallup’s heroic assumption of an electorate that would look racially more similar to 2000 than 2008 doomed their overall forecast.
The graph below shows the 1984 to 2012 trend in the proportion of 18-29 year olds in the active electorate and the Gallup projection for young adults in 2012. Considering the trend demonstrates that Gallup’s youth composition estimate of 13 percent was just as eccentric as their non-Hispanic white projection. Since 1984, youth voters never fell below 17 percent of the active electorate. And the recent trend shows great stability, from 1996-2008 young voters comprised either 17 or 18 percent of the active electorate in each presidential election. The 2012 exit polls indicate that 18-29 year olds made up 19 percent of voters, 6 points higher than Gallup’s estimate. Because young voters gave Obama 60 percent of their vote in 2012, and no other age group gave Obama more than 52 percent (30-44yrs, 52%) (45-64yrs, 47%), (65+yrs, 44%) Gallup’s underestimation of youth voters also skewed Gallup’s overall expectations about the winner of the 2012 contest.
That the outlier poll (Gallup) was the most inaccurate survey in 2012 should not come as a surprise to Monkey Cage readers. What is perhaps surprising is that Gallup simply would have needed to review basic voter composition trends by key age groups and racial categories to see the obvious sources of error in their assumptions. One view might praise Gallup for not adjusting their longstanding likely voter formula to better conform to conventional wisdom or even past trends. However, moving forward, it seems that Gallup’s likely voter formula will continue to systematically underestimate the proportion of young adults and non-white voters. As non-whites and young voters increased in size as a proportion of the electorate and became more distinctive relative to the overall voting population, underestimating the proportion of non-whites and young adults in the active electorate has become more consequential for Gallup’s overall estimates. Think of this, just 12 years ago 18-29 year olds split their vote evenly between Kerry and G.W. Bush (51% to 49%) and non-Hispanic whites still comprised over 80 percent of voters. Times have changed and so should Gallup’s system for determining likely voters.