The Constant Attention to Individual Polls Is Hurting America

This morning, Nate Silver tweeted:

Retweeting individual poll results is probably disinformative on balance.

Which I gladly retweeted.  And then came the new Pew survey.  I’m sorry, I mean THE NEW PEW SURVEY.  The collective reaction of political commentators can be summarized with the frontpage of the Huffington Post:

 

(And HuffPo is not alone.)  For my part, I noted that it appeared Silver’s admonition would go unheeded.  Silver, for his part, grew even more cranky.

Let me say why this constant attention to the latest poll—if not quite living up to the deliberate hyperbole of this post’s title—is pathological nevertheless.

1) Uncertainty, uncertainty, uncertainty.  Poll results vary for random reasons—that is, because of sampling error.  Everyone “knows” this, but to see headlines like HuffPo’s, you’d think that Pew was The One Blessed Holy Gospel Account of American Public Opinion.  However, we don’t actually know the true proportions of Obama and Romney supporters in the public.  There is no way to know definitively which poll is “the truth.”  We have some sense of which polls have smaller or larger house effects in this cycle—i.e., which ones are farther from or closer to the average of the polls—but, as Drew Linzer pointed out, the estimates of house effects also come with substantial uncertainty themselves.  So not only don’t we know Obama’s or Romney’s true share of the vote at this moment, but we don’t know with much confidence how much any pollster might be systematically overestimating or underestimating Obama’s or Romney’s share of the vote.  For that reason, if you just want to know where the horserace stands, look at the average of the polls and ignore the individual polls.

2) The polls that are the most “surprising” are often the least diagnostic.  One possible reason to pay attention to Pew is that it appears to have a Democratic lean, so if it finds Romney on top, that must be newsworthy.  Let’s assume Pew has a Democratic lean.  If so, it’s possible that the Pew result is significant in this way, but I think the opposite is more likely: results like this one are outliers.  Indeed, this Pew survey is out of line with most other polls conducted since the debate.  So rather than being more diagnostic, it is less.  (Note that this is not a slam on Pew, which is a very scrupulous pollster.  But the kinds of random variation I’m talking about having nothing to do with the quality of a pollster.)

3) Tweeting every individual poll encourages the amateur Talmudists.  Once people’s attention is drawn to some individual poll, then they start picking it apart.  How many Democrats and Republicans were in the sample?  What percent of blacks supported Obama?  Did it sample registered voters or likely voters?  How could Obama be ahead by this much in such-and-such state but behind by that much in some other state?  IS THE POLL SKEWED OR UNSKEWED? And so on and on and on.  Such questions might occasionally illuminate flaws in a poll, even though the party composition of the sample is not some gold standard for a poll’s quality.  But the problem is this: the kinds of “lessons” people learn by delving into a poll are even more tenuous because the uncertainty that applies to the toplines applies in greater measure to the cross-tabs.  What is the margin of error on any nationally representative poll’s estimate of the views of men, women, whites, blacks, Jews, left-handed college professors, Southerners, etc., etc.?  It is large.  Yes, everyone “knows” this too.  But that doesn’t stop the Talmudists from weighing on a poll’s quality using criteria that are sometimes misguided and with a degree of certainty that is almost always unwarranted.

4) Tweeting every individual poll unnecessarily reduces our confidence in polls.  The polls this year are well-behaved.  Let me repeat: they are well-behaved.  Go read Drew Linzer’s post at the link.  This is far from the impression that people have, however.  Why?  When commentators fixate on every individual poll—and especially on the polls that are far from the average and therefore surprising—they give the misleading impression that the polls are “all over the place” or “crazy” or “inexplicable.”  And this makes some voters suspicious.  They think that polls suck.  They think that pollsters are manipulating the results.  After all, why are they getting such different answers?  Here’s one example from someone replying to my earlier tweet:

Aug. Pew poll was 6 pts out of whack pro-Obama. This one 6 pts. out too. Like ump making up for blown call.

Of course, that’s not what Pew was doing.  But this comment manifests the skepticism that pollsters confront.  Although pollsters shouldn’t be treated like oracles, they don’t deserve these conspiracy theories either.

I’m not suggesting that everyone who tweets a poll result is unaware of these points.  But I am also well-aware of the incentives of news organizations to find the newest, shiniest thing and shout about it.

So even though I don’t expect this little blog post to make any difference, we’d truly have a much better conversation about this election if people would stop fixating on every single poll.

18 Responses to The Constant Attention to Individual Polls Is Hurting America

  1. Andrew Gelman October 8, 2012 at 8:25 pm #

    John:

    I agree completely. When we wrote our 1993 paper, Gary and I naively believed this would eb enough to convince news organizations to stop running so many polls. But no….

    On a related topic, though, are we at the Monkey Cage contributing to the problem by posting links to people’s poll-based election forecasts, for example this map based on September state-level polls? This just adds to the noise, no?

    • Andrew Gelman October 8, 2012 at 8:30 pm #

      P.S. I don’t mean to pick on that particular post I linked to, which I’m sure has value. I’m just thinking more generally that our many posts–including my own!–about polls can be in some way contributing to the problem. It’s perhaps unavoidable, as people are talking about the polls whether we discuss them or not, but still it seems like an issue.

    • John Sides October 8, 2012 at 8:34 pm #

      Andy: I agree that it’s more useful to look at combinations of forecasts than any single forecast — and I think I’ve tried to make that point from time to time. I would have been more reticent about linking to that one map, except it’s broadly in line with a bunch of others — 538, Pollster, Votamatic, etc. — and so I didn’t feel like I was chasing after an outlier in order to generate clicks. More like I was adding another data point to the current consensus.

    • Tom Holbrook October 9, 2012 at 9:58 pm #

      Hey, I kind of like that map. Actually, the forecasts reported in that map are based on hundreds of poll–in most cases several for each state. Beyond that, the model also includes past voting trends and an average of national polls. Since John’s point was about the problem of jumping to new conclusions every time a new poll comes out, I’m not sure this forecast is part of the problem

  2. Jonathan October 8, 2012 at 9:09 pm #

    Agreed 100% John….with a thought. Isn’t it possible that amateur talmudists in this case can help point out that polls like Pew are outliers by showing how irregular the gender gap in the poll is etc…

    • John Sides October 8, 2012 at 10:04 pm #

      That would work as long as the uncertainty in the cross-tabs doesn’t wash out our ability to make comparisons across polls. Which it often does…

  3. Donald October 8, 2012 at 9:32 pm #

    Interesting that you’d criticize the focus on a single poll after THIS poll. Your argument might have been better received — or Nate’s — had you been making it when O was nine points up in the last Pew survey.

  4. Ryan Enos October 8, 2012 at 9:46 pm #

    John,
    completely agree. Serious question that I worry about sometimes though: when everybody starts looking at poll averages, rather than individual polls, do we kill the incentive to conduct individual polls accurately, if at all, thus hurting the accuracy of the averages? Maybe a little attention to polls once and while keeps them going? I suppose that most polling houses make their money with consumer polls, but the publicity or political polls helps, right? Something needs to keep that publicity going…
    Ryan

    • John Sides October 8, 2012 at 10:00 pm #

      Ryan: I don’t disagree. The problem is the current incentives encourage pollsters to trumpet any result that seems new or different. (Again, not suggesting that Pew or any pollster is negligent here.) A focus on averages — which is what reporters, commentators, and lay readers can do themselves — reduces that incentive on the margins.

      I think a positive incentive for accurate polling is going to come more from professional standards and pressure to abide by those standards — especially with regard to transparency. Although I don’t know how effective any such standards will be, since there seems to be no penalty for ignoring them.

      Which then leads me back to thinking that the consumers of the polling data can best apply any pressure here.

    • Andrew Gelman October 8, 2012 at 10:10 pm #

      Ryan:

      I’d love to “kill the incentive to conduct individual polls accurately.” We could reduce the number of individual polls to approximately zero, if all that’s going to be done with them is reporting toplines.

  5. John in Hollywood October 9, 2012 at 2:00 am #

    Nate Silver makes a living commenting on polls every single day, so I think he should probably STFU or admit that he is a part of the nations current obsession with polls.

    • Andrew October 9, 2012 at 11:29 am #

      Way to miss the entire point of the post, John in Hollywood.

      No one is suggesting that the problem here is “the nation[‘]s current obsession with polls.” The problem is the intense focus on single (usually outlier) polls rather than the polling average. The media has a strong incentive to focus on outliers because they appear “newsworthy.” But, as Prof. Sides notes, these polls are by their very nature the LEAST valuable in terms of diagnostic or predictive value.

      Nate Silver’s aggregation of polls (just like those of HuffPo, RCP, etc.) does us a service by encouraging people to focus on the entire set of data rather than just a single point.

    • High Plains Drifter October 9, 2012 at 11:52 am #

      I bet Nate takes weekends off sometimes. I think the post above is criticizing obsession with outlying top numbers that have been drained of their true *uncertain* character. Simply calling it “obsession with polls” lumps that behavior in with the kind of interest that includes understanding the quality of knowledge the polls offer. That can be interesting even if that knowledge is uncertain.

      As a news consumer, I am less concerned about tweets than I am about the reporting practice that almost universally drops the errors from the reporting. I don’t think any reputable news organization should report an individual poll without reporting the errors in the same sentence (or box, etc. nearby!).

  6. Raoul October 9, 2012 at 1:22 pm #

    The issue I find more interesting in this latest dust-up is a possible challenge to what I perceived as the conventional wisdom that “the debates don’t matter much” on the final election outcome.

    I see the distinct possibility that debates do matter generally and are will be increasing in importance:

    1) The scripted convention speeches are no longer a big event. Obama had 35 million viewers and Romney 30M whereas the debate had close to 70M (nearly all of whom will vote). That 70M represents well over 50% of the likely votes cast in the 2012 election. I would posit that the erosion in convention importance is actually increasing the importance of the debates. (It would be interesting to compare TV ratings over time between convention speeches and debates – I presume a gap is growing)

    2) Debate is reality TV and Americans not only like reality TV, they take it very seriously. Shows like The Voice, Dancing With The Stars and American Idol have trained Americans to watch performances side by side and judge for themselves – often over a period of weeks. The phenomenon was not as evolved in 2008 but it is highly evolved now, and the popularity of those shows attests to this. The best news for Obama in this is that folks have been trained to treat such evaluations in a serial fashion such that they will give Obama more than one chance. But if he blows it again, then he will be in serious trouble and Andrew Sullivan’s hair-on-fire tweets may become reality.

    I imagine it is still too early to discern a trend from the post debate polling, and I agree with the need to focus on numerous polls, not one. But as a regular person I can tell you one thing, talk around the water cooler about this debate was unusual compared to prior years and was consistent with the kind of focus I see on American Idol and DWTS. People care about reality TV contests and they will incorporate that information in their judgments on an increasing basis. Not sure if that is good or bad, but I think it means the debates will matter and I think subsequent polls will show that.

  7. LFC October 9, 2012 at 9:02 pm #

    Fyi, the Pew poll also got attention on the PBS NewsHour, where Andrew Kohut (of Pew) is a regular guest. There was no mention of margins of error and no mention, iirc, of other polls in that particular discussion.

  8. LFC October 9, 2012 at 9:20 pm #

    I just read the Andrew Sullivan post that J. Sides linked to. Sullivan’s post is a disgrace — not only in its single-minded focus on one poll but in its hyperbolic, over-the-top and unwarranted assertion that Obama was “incapable of making a single argument.” He didn’t make the arguments as crisply or effectively as he might have, but he did make some arguments. It wouldn’t matter if no one read Sullivan, but people read him. He blogs for a living. The notion that Sullivan is getting paid to write a piece of garbage like this speaks volumes about the state of online commentary. If I were his employer I would fire him on the basis of this post alone.