The Constant Attention to Individual Polls Is Hurting America

by John Sides on October 8, 2012 · 18 comments

in Campaigns and elections,Public opinion


Andrew Gelman October 8, 2012 at 8:25 pm


I agree completely. When we wrote our 1993 paper, Gary and I naively believed this would eb enough to convince news organizations to stop running so many polls. But no….

On a related topic, though, are we at the Monkey Cage contributing to the problem by posting links to people’s poll-based election forecasts, for example this map based on September state-level polls? This just adds to the noise, no?

Andrew Gelman October 8, 2012 at 8:30 pm

P.S. I don’t mean to pick on that particular post I linked to, which I’m sure has value. I’m just thinking more generally that our many posts–including my own!–about polls can be in some way contributing to the problem. It’s perhaps unavoidable, as people are talking about the polls whether we discuss them or not, but still it seems like an issue.

John Sides October 8, 2012 at 8:34 pm

Andy: I agree that it’s more useful to look at combinations of forecasts than any single forecast — and I think I’ve tried to make that point from time to time. I would have been more reticent about linking to that one map, except it’s broadly in line with a bunch of others — 538, Pollster, Votamatic, etc. — and so I didn’t feel like I was chasing after an outlier in order to generate clicks. More like I was adding another data point to the current consensus.

Tom Holbrook October 9, 2012 at 9:58 pm

Hey, I kind of like that map. Actually, the forecasts reported in that map are based on hundreds of poll–in most cases several for each state. Beyond that, the model also includes past voting trends and an average of national polls. Since John’s point was about the problem of jumping to new conclusions every time a new poll comes out, I’m not sure this forecast is part of the problem

Jonathan October 8, 2012 at 9:09 pm

Agreed 100% John….with a thought. Isn’t it possible that amateur talmudists in this case can help point out that polls like Pew are outliers by showing how irregular the gender gap in the poll is etc…

John Sides October 8, 2012 at 10:04 pm

That would work as long as the uncertainty in the cross-tabs doesn’t wash out our ability to make comparisons across polls. Which it often does…

Donald October 8, 2012 at 9:32 pm

Interesting that you’d criticize the focus on a single poll after THIS poll. Your argument might have been better received — or Nate’s — had you been making it when O was nine points up in the last Pew survey.

John Sides October 8, 2012 at 9:53 pm

Donald: I’ve been making a species of this point forever, so please spare me your insinuations of partisan bias. Here, based a quick search, are three other posts in this vein, one criticizing emphasis on a poll that exaggerated the positive effects of a Bill Clinton speech, one criticizing emphasis on polls that exaggerated a surge for John McCain in 2008, and one criticizing emphasis on a poll that exaggerated the effects of one of Obama’s speeches (with a bonus slam for exaggerating an increase in Obama’s approval):

Nate Silver October 9, 2012 at 1:06 am

On Aug. 2, I wrote a piece criticizing the focus on the Pew poll, which at that point showed a 10-point Obama lead.

Ryan Enos October 8, 2012 at 9:46 pm

completely agree. Serious question that I worry about sometimes though: when everybody starts looking at poll averages, rather than individual polls, do we kill the incentive to conduct individual polls accurately, if at all, thus hurting the accuracy of the averages? Maybe a little attention to polls once and while keeps them going? I suppose that most polling houses make their money with consumer polls, but the publicity or political polls helps, right? Something needs to keep that publicity going…

John Sides October 8, 2012 at 10:00 pm

Ryan: I don’t disagree. The problem is the current incentives encourage pollsters to trumpet any result that seems new or different. (Again, not suggesting that Pew or any pollster is negligent here.) A focus on averages — which is what reporters, commentators, and lay readers can do themselves — reduces that incentive on the margins.

I think a positive incentive for accurate polling is going to come more from professional standards and pressure to abide by those standards — especially with regard to transparency. Although I don’t know how effective any such standards will be, since there seems to be no penalty for ignoring them.

Which then leads me back to thinking that the consumers of the polling data can best apply any pressure here.

Andrew Gelman October 8, 2012 at 10:10 pm


I’d love to “kill the incentive to conduct individual polls accurately.” We could reduce the number of individual polls to approximately zero, if all that’s going to be done with them is reporting toplines.

John in Hollywood October 9, 2012 at 2:00 am

Nate Silver makes a living commenting on polls every single day, so I think he should probably STFU or admit that he is a part of the nations current obsession with polls.

Andrew October 9, 2012 at 11:29 am

Way to miss the entire point of the post, John in Hollywood.

No one is suggesting that the problem here is “the nation[']s current obsession with polls.” The problem is the intense focus on single (usually outlier) polls rather than the polling average. The media has a strong incentive to focus on outliers because they appear “newsworthy.” But, as Prof. Sides notes, these polls are by their very nature the LEAST valuable in terms of diagnostic or predictive value.

Nate Silver’s aggregation of polls (just like those of HuffPo, RCP, etc.) does us a service by encouraging people to focus on the entire set of data rather than just a single point.

High Plains Drifter October 9, 2012 at 11:52 am

I bet Nate takes weekends off sometimes. I think the post above is criticizing obsession with outlying top numbers that have been drained of their true *uncertain* character. Simply calling it “obsession with polls” lumps that behavior in with the kind of interest that includes understanding the quality of knowledge the polls offer. That can be interesting even if that knowledge is uncertain.

As a news consumer, I am less concerned about tweets than I am about the reporting practice that almost universally drops the errors from the reporting. I don’t think any reputable news organization should report an individual poll without reporting the errors in the same sentence (or box, etc. nearby!).

Raoul October 9, 2012 at 1:22 pm

The issue I find more interesting in this latest dust-up is a possible challenge to what I perceived as the conventional wisdom that “the debates don’t matter much” on the final election outcome.

I see the distinct possibility that debates do matter generally and are will be increasing in importance:

1) The scripted convention speeches are no longer a big event. Obama had 35 million viewers and Romney 30M whereas the debate had close to 70M (nearly all of whom will vote). That 70M represents well over 50% of the likely votes cast in the 2012 election. I would posit that the erosion in convention importance is actually increasing the importance of the debates. (It would be interesting to compare TV ratings over time between convention speeches and debates – I presume a gap is growing)

2) Debate is reality TV and Americans not only like reality TV, they take it very seriously. Shows like The Voice, Dancing With The Stars and American Idol have trained Americans to watch performances side by side and judge for themselves – often over a period of weeks. The phenomenon was not as evolved in 2008 but it is highly evolved now, and the popularity of those shows attests to this. The best news for Obama in this is that folks have been trained to treat such evaluations in a serial fashion such that they will give Obama more than one chance. But if he blows it again, then he will be in serious trouble and Andrew Sullivan’s hair-on-fire tweets may become reality.

I imagine it is still too early to discern a trend from the post debate polling, and I agree with the need to focus on numerous polls, not one. But as a regular person I can tell you one thing, talk around the water cooler about this debate was unusual compared to prior years and was consistent with the kind of focus I see on American Idol and DWTS. People care about reality TV contests and they will incorporate that information in their judgments on an increasing basis. Not sure if that is good or bad, but I think it means the debates will matter and I think subsequent polls will show that.

LFC October 9, 2012 at 9:02 pm

Fyi, the Pew poll also got attention on the PBS NewsHour, where Andrew Kohut (of Pew) is a regular guest. There was no mention of margins of error and no mention, iirc, of other polls in that particular discussion.

LFC October 9, 2012 at 9:20 pm

I just read the Andrew Sullivan post that J. Sides linked to. Sullivan’s post is a disgrace — not only in its single-minded focus on one poll but in its hyperbolic, over-the-top and unwarranted assertion that Obama was “incapable of making a single argument.” He didn’t make the arguments as crisply or effectively as he might have, but he did make some arguments. It wouldn’t matter if no one read Sullivan, but people read him. He blogs for a living. The notion that Sullivan is getting paid to write a piece of garbage like this speaks volumes about the state of online commentary. If I were his employer I would fire him on the basis of this post alone.

Comments on this entry are closed.

Previous post:

Next post: