Gallup(ing) Away from the Herd

by John Sides on November 7, 2012 · 18 comments

in Campaigns and elections

This is a guest post by Samuel Best, Associate Professor of Political Science at the University of Connecticut, and Brian Krueger,Professor of Political Science at the University of Rhode Island. The information about voters in this post was drawn from Gallup, CNN, and dimpledchad.info, which is a new website launched by the authors and Jeffrey Ladewig and Clifford Vickrey. Using information derived from U.S. Census surveys and national exit polls, Dimpled Chad displays the behavior of a wide range of specialized demographic groups in federal elections spanning as far back as 1972.

 *****

The American public has unprecedented access to the results of public opinion polls.  Indeed, if individuals spend any time reading, listening, or watching news, they could hardly escape reports of the latest measurements of the public mind. Almost a century ago, theoretical physicist Werner Heisenberg taught us that we cannot measure something without simultaneously disturbing the object of our measurement. Increasingly, campaigns, media surrogates, and pundits agree that public opinion polls don’t just measure public opinion, these polls also shape preferences and influence behaviors ranging from turning out to vote to donating to campaigns.  Accordingly, favorable polls get lauded and unfavorable polls dismissed as flawed or purposively biased.

Since Mitt Romney wrapped up the nomination, Barack Obama has led in most national polls.  Predictably, supporters of the Republican ticket questioned the accuracy of the polls.  During the fall campaign, these poll doubters were often dismissed as anti-scientific, Republican wishful thinkers, conspiracy theories, or base partisan hacks.  Nate Silver became so frustrated at the broadside against the polls that he even harnessed his inner Mitt Romney and bet thousands of dollars that the poll averages would prevail in predicting the election winner.  For Silver, only systematic bias across the many survey houses could lead to a Romney victory.

But one venerable polling firm, Gallup, offered the poll doubters hope, by consistently reporting that likely voters supported Romney over Obama, sometimes by wide margins.  Karl Rove’s Wall Street Journal opinion piece and Josh Gordon’s National Review article published in November 2012 use Gallup’s results to predict a Romney win.  The argument is straightforward.  Gallup, unlike every other major, non-partisan polling outfit, anticipates that those who identify as Republicans will comprise a greater portion of the electorate than Democrats.  And because about 90 percent of both Democrats and Republicans vote for their party’s nominee, small shifts in the partisan balance of the sample create substantial changes in the overall estimate. If Gallup is correct (and everyone else is wrong) about the partisan composition of the 2012 active electorate, it would represent the type of systematic bias that Nate Silver warns could sink the usually sound predictions from poll aggregation.

Fortunately, and to Gallup’s great credit, a few days ago they reported their final expectations about the partisan make-up of the 2012 active electorate.   It should be noted that Gallup does not weight their sample based on a previously identified parameter about the distribution of partisans in the 2012 active electorate.  Instead, Gallup uses a likely voter formula that identifies which respondents are likely to vote and which ones are not likely to vote.  If more Republicans than Democrats happen to be identified as likely voters, then Gallup will have more Republicans in their likely voter sample.  This year, Gallup expects the active electorate to comprise 36 percent Republicans, 35 percent Democrats, and 29 percent Independents (including leaners).  A review of eight other major polls revealed that none showed a Republican partisan advantage, with an average Democratic partisan advantage of 4 points.

Now that the exit polls are public, we can assess Gallup’s expectations about a Republican partisan advantage, the lynchpin of Gallup’s lonely prediction that Romney would win the popular vote.  The exit polls, while having much higher response rates than conventional polls, and remain our best guess about the composition of the active electorate, are not perfect.  Exit polls suffer from the uncertainty and potential bias inherent in any survey.  Exit polls use both phone surveys to estimate early voting and a more traditional stratified probability sample of voters leaving their polling place.  Each type of sample is combined (totaling more than 20,000 cases) and weighted with actual election results. Because exit polls use a sample of all voters, the results will not flawlessly represent all voters.  Even so, because of the high relative response rate, the ability to regularly directly identify voters, and the advantage of weighting based on actual vote returns, the exit polls are unmatched as a tool for describing the voting population.

The graph below shows the 1984 to 2012 exit poll trends in the partisan composition of the active electorate and the Gallup partisan composition projection for 2012.  Happily, the exit polls, just like Gallup, report partisanship by excluding independents who lean toward a party.  The circles represent Gallup’s projections about the partisan composition of the electorate and the trend line is the historic partisan composition of the active electorate according to the exit polls.  Gallup expected the active electorate to contain 35 percent Democrats, 36 percent Republicans and 29 Percent Independents.  Gallup’s two partisan projections missed badly. Democrats comprised about 38 percent of the exit polls, which was in line with their multi-decade average and 3 points higher than Gallup’s projection.  The partisan composition trend makes clear that 2004, not 2008, was the outlier election year for Democrats.  Republicans comprised just 32 percent of the exit polls, about 4 points less than Gallup’s expectation (36 percent).  The 32 percent Republican share of the electorate matched 2008, but was down from the peak in 2004 (37 percent) as well as the multi decade average of 35 percent.  Again 2004 appears to be the most different case from 2012, not 2008.  Because partisans overwhelmingly supported their party’s candidate (over 90 percent for both), Gallup’s expectation that more Republicans would cast ballots than Democrats doomed their overall election prediction.  It is worth noting that the multi-poll-average partisan composition advantage of 4 points nailed it (see above).  Chalk up another win for poll aggregation.


We know that Gallup did not force their likely voter sample to fit a pre-determined distribution of partisan identification.  What then led the Gallup likely voter formula to oversample Republicans?  Some clues are found when also considering Gallup’s projections about the proportion of women, young adults, and whites in the active electorate.  The graph below shows the trend in the proportion of women among voters from 1984 to 2012 and Gallup’s projection about the proportion of women in the 2012 active electorate.  Gallup expected 52 percent of voters to be women, whereas the exit polls suggest 53 percent of 2012 voters were women.  Gallup’s expectation about the proportion of women in the active electorate does not appear to be the prime culprit explaining Gallup’s error.  Gallup missed by just a point and women were largely split in their vote, giving 55 percent to Obama.  Replacing 1 percent of men, who cast 45 percent for Obama, with women would have scarcely moved the overall Gallup estimate.


The graph below shows the trend in the proportion of non-Hispanic whites in the active electorate from 2000 to 2012 and the Gallup projection about the size of the non-Hispanic white voting bloc in 2012.  The exit poll race question is only consistent with Gallup over the past four elections.  Accordingly, the trend goes back as far as 2000.  Gallup estimated that non-Hispanic whites would comprise 78 percent of voters in 2012.  This would have been the highest proportion of whites in a presidential active electorate since 2000.  Given the secular downward trend in white composition, this Gallup expectation never had a chance.  In 2012 the exit polls indicate that 72 percent of voters were white, 2 points less than 2008 and a full 6 point lower than Gallup’s expectations.  Because about 80 percent of non-whites cast ballots for Obama compared to 39 percent of non-Hispanic whites, Gallup’s 6 point racial composition gap explains much of their overall error.  Simply put, Gallup’s heroic assumption of an electorate that would look racially more similar to 2000 than 2008 doomed their overall forecast.


The graph below shows the 1984 to 2012 trend in the proportion of 18-29 year olds in the active electorate and the Gallup projection for young adults in 2012.  Considering the trend demonstrates that Gallup’s youth composition estimate of 13 percent was just as eccentric as their non-Hispanic white projection.  Since 1984, youth voters never fell below 17 percent of the active electorate.  And the recent trend shows great stability, from 1996-2008 young voters comprised either 17 or 18 percent of the active electorate in each presidential election.  The 2012 exit polls indicate that 18-29 year olds made up 19 percent of voters, 6 points higher than Gallup’s estimate.  Because young voters gave Obama 60 percent of their vote in 2012, and no other age group gave Obama more than 52 percent (30-44yrs, 52%) (45-64yrs, 47%), (65+yrs, 44%) Gallup’s underestimation of youth voters also skewed Gallup’s overall expectations about the winner of the 2012 contest.

That the outlier poll (Gallup) was the most inaccurate survey in 2012 should not come as a surprise to Monkey Cage readers.  What is perhaps surprising is that Gallup simply would have needed to review basic voter composition trends by key age groups and racial categories to see the obvious sources of error in their assumptions.  One view might praise Gallup for not adjusting their longstanding likely voter formula to better conform to conventional wisdom or even past trends.  However, moving forward, it seems that Gallup’s likely voter formula will continue to systematically underestimate the proportion of young adults and non-white voters.  As non-whites and young voters increased in size as a proportion of the electorate and became more distinctive relative to the overall voting population, underestimating the proportion of non-whites and young adults in the active electorate has become more consequential for Gallup’s overall estimates.  Think of this, just 12 years ago 18-29 year olds split their vote evenly between Kerry and G.W. Bush (51% to 49%) and non-Hispanic whites still comprised over 80 percent of voters. Times have changed and so should Gallup’s system for determining likely voters.

 

{ 18 comments }

Sam Popkin November 7, 2012 at 3:01 pm

Excellent post. Hopefully one more crack in the the pedestal from which Gallup towers over other pollsters in public imagination.
How long will the Gallup Brand name continue to persuade innocent bystanders that they are “The” poll? At a point when five tracking polls had a virtual tie and nine national polls averaged Obama by 0.2 percent, Nate Silver calculated that the average newspaper story had Romney by four percentage points thanks to the concentration on Gallup in the press.

George Gallup was a brilliant salesman and a pioneer. He was never a pioneer on sampling or psychology. He also stopped polling early in October 1948 since he “knew” that voters didn’t reconsider their decisions.

tom russell November 29, 2012 at 2:30 pm

gallup was skewed but Americans usually do not follow polls , we are the poll that counts. Pollsters avg of polls was right on , and 538 with Silver was surprising right . The best rule is to use the wealth of info on the net without bias , a difficult skill indeed.

Jimbo November 7, 2012 at 3:02 pm

Let’s face facts: like Rasmussen, Gallup is a partisan polling firm seeking to push an agenda. They are totally discredited.

Some Body November 8, 2012 at 6:34 am

Evidence?

Seriously, this response sounds like something Joe Scarborough or Karl Rove would say, only for the other party.

Peter Principle November 10, 2012 at 3:53 pm

Let’s just say I would be curious to hear an alternative explanation for why Gallup assumed the whitest electorate since 2000, and the oldest probably in American history — when virtually everyone EXCEPT the GOP and the Romney campaign was telling them they were making big mistakes.

So far, they haven’t provided one.

Occam’s razor, anyone?

LTB666 November 14, 2012 at 8:47 pm

Add to that the fact that their last poll (R+1) was mysteriously much more in line with what other pollsters said than the one before that (R+5!).
They were still wrong but being wrong is not as bad as adjusting your polls in the last minute to not make you look THAT stupid.

Bill November 7, 2012 at 3:06 pm

Wow, those plots are amazing. Gallup’s (presumably implicit) assumption that the non-white proportion would shoot back up after steady past decline seems like malpractice, especially given how many analysts have emphasized the growing Hispanic population. Their expected sudden drop in young voters seems similarly implausible given prior trends.

Somehow I think Gallup will wake up to these issues by 2016, if only to avoid humiliation.

mz November 19, 2012 at 4:19 am

my goodness, I am just a very interested layman, one of those legions hooked on the science of analyzing the science of polling through Nate Silver, Mark Blumenthal, Sam Wang and when feeling fearless enough, following the articles at Monkey Cage.

I must have read 5 different pieces from folks like Alan Abramowitz and Mark Blumenthal basically shouting from the rafters that Gallup was woefully underrepresenting non-whites and the young in its models. I am SURE the Newport crew knew of this controversial prediction, that somehow white turnout was going to INCREASE 4 % in defiance of all trends the past 30 years, but they stuck to their guns.

They were publicly called on the carpet in the most widely-read and influential media organ in the country, thanks to 538 in the NY Times, first in the famous Gallup against the World piece and then in the post mortem when they were shamefully rated as the worst and most GOP biased pollster by 3.5%

Their reliability has taken a terrible blow, and if they do not want to be relegated to being seen as a pro-GOP organ, I have to hope they clean up their act. You do have to wonder if money changed hands. I am not a conspiracist by nature, but the mockery and discredit they received as part of their woefully anti-science demographic projection just makes no sense. If they were working for the Romney campaign in a subterranean way, then it would make sense, because otherwise, it just was not worth the blow to their reputation to be so stubborn in their anti-science demography.

Mike S November 7, 2012 at 3:12 pm

Gallup was reflected in several news outlets as more “pure” because it doesn’t apply all that scienc-y stuff to control for known sampling biases. Good article, except for perpetuating the incorrect description of Heisenberg’s uncertainty principle!

Carole November 7, 2012 at 3:25 pm

This is great. It does help explain a lot. But these disparities between the expected and observed demographic composition is driven by a set of assumptions that still seem to be a bit of a mystery. I understand that past behavior and commitment are components, but I think a little more is needed. I’m still unsure as to what exactly goes into Gallup’s LV formula and how it varies from the screens other polling companies were using towards the end of the campaign. Apart from Gallup’s status as an outlier compared to other polling firms, within their own numbers, it might have been apparent from the size of the gap between the registered voter candidate choice and those for likely voters that something might be amiss.

Dharma November 7, 2012 at 5:20 pm

Great post, excellent analysis. Have been looking for this explanation and am glad to finally find it. I read the comments above, some of which speak to Gallup’s potential “motivation” but it was clear that manipulation, intended or unintended, was going on with the Gallup poll after the first debate. The lack of creditability, usefulness, and impartiality will be very difficult for Gallup to overcome.

idiot November 8, 2012 at 10:15 am

I sincerely hope that Gallup’s reputation recovers and that it changes its voting system so as to be more accurate in the future. I don’t trust unfamiliar polling outlets, outwardly partisan outlets like PPP (D), or covertly partisan outlets like Rasmussen. And I don’t want to rely on modelling, because such modelling require some actual basis in popular opinion, meaning surveys.

grandpa john November 8, 2012 at 8:54 pm

PPP was tied for First in poll accuracy ratings. Gallup and Rasmussen were in the bottom 5 of the around 30 pollsters rated.

Travis R November 8, 2012 at 11:46 pm

Here is a good breakdown of swing states via PPP and Rasmussen. PPP did very well this year, as they did in 2008.

http://travistation.wordpress.com/2012/11/08/a-fun-post-mortem-of-the-polls/

counsellorben November 8, 2012 at 4:08 pm

Technical note: you refer to the Kerry-Bush election of 12 years ago. Were you intending to refer to the Bush-Gore election of 12 years ago, or the Kerry-Bush election of 8 years ago?

Travis R November 8, 2012 at 6:41 pm

There was a very obvious and severe flaw that caused Gallup to grossly undercount young voters. I detailed it here back on October 19: http://travistation.wordpress.com/2012/10/19/gallup-doesnt-think-im-going-to-vote-because-im-a-flaky-young-whippersnapper/

Essentially, I am a hyper-partisan liberal who drove every friend and family member crazy the past 6 months or so. Yet Gallup said I wasn’t even going to vote, and it wasn’t even close. (I rated only a 4 out of a 7 point “likely voter” scale) See the blog for details.

Eric LeVine November 9, 2012 at 2:39 pm

In the aftermath, it’s sort of funny that the Obama pollster was 100% accurate in calling out Gallup on these exact issues in mid-October.

http://thecaucus.blogs.nytimes.com/2012/10/15/now-democrats-suggest-polling-is-flawed/

Peter Principle November 10, 2012 at 3:58 pm

That blog of all the white people mourning Romney’s defeat ought to include a picture of Frank Newport.

Comments on this entry are closed.

Previous post:

Next post: