“Charlie Cook”:http://www.nationaljournal.com/njonline/po_20100928_1190.php?mrefid=email_offtotheraces:
bq. More than in any previous cycle that we’ve witnessed, perceptions of the ebb and flow of races are being driven by state- and district-level polling. This does not mean that there is better polling, just more. To some, polling has become a commodity — all the same and totally interchangeable. This is pure rubbish.
bq. Probably 90 percent of the public polling in statewide and district races is mediocre at best, and much of it is very close to worthless.
bq. Despite this, it is still driving the narrative of where individual races stand. Some of these polls screen for likely voters — which is important in lower-turnout midterm elections — while others don’t bother to screen at all. Some don’t seem to bother weighting for gender, age and other predictable variables. Some include the names of all the candidates on the ballot and others do not…
bq. …My view is that most academic polling, as well as the polling sponsored by local television stations and newspapers, is dime-store junk.
bq. The far more sophisticated polling is done by top-notch professional polling firms for campaigns, parties and major business and labor organizations. These polls are considerably more expensive and the methodology is more rigorous.
bq. Most of these surveys are not made public, but insiders can be made aware of them. While even the most experienced and contentious political pollsters have more challenges than a generation ago, their work is still far superior and reliable.
bq. The end result is that you have two separate conversations about these political races: one that is driven by the publicly available, but less reliable, stock of polls and the other made by the black market of high-quality and more expensive surveys done for private clients, including the campaigns themselves.
I am mystified. Let’s assume that our focus is on pollsters doing pre-election polling and that we should measure the quality of these polls based on how well they predict election results. This isn’t remotely the entire story about polling, but these are the pollsters and polls that Cook is talking about.
First, it is an exaggeration to call “90%” of public polling “mediocre” and “much of that…worthless.” This is simply at odds with actual data-driven assessments of poll quality. Nate Silver’s “assessment”:http://www.fivethirtyeight.com/search/label/pollster%20ratings of the accuracy of pre-election polls finds that, with one exception, the best and worst pollsters were only 1.8 points apart in terms of “pollster-induced error.” David Shor’s “assessment”:http://stochasticdemocracy.blogspot.com/2010/06/how-to-detect-bad-pollster-and-more.html reached a similar conclusion. In other words, there simply isn’t enough difference among pollsters to justify claims like “mediocre” and “worthless.”
Second, there is this slap at “academic polling” as “dime-store junk.” I mean, let’s name names here. What academics or academic institutions are doing worthless pre-election polling?
Third, there are the claims about private polls commissioned for political actors. I agree with Cook that there’s no reason to dismiss these polls on methodological grounds. However, absent some proof, I doubt that they are “far more sophisticated” or “more rigorous” or “more superior and reliable”. I am sure you can construct some comparison of certain public polls and certain private polls where these claims might hold true. But these broad generalizations don’t hold water. Cook should discuss what exactly about these polls makes them so much more sophisticated, how he assesses their reliability and rigor, etc. I think it might also be valuable to distinguish among public polls as well. Certainly the methodologies of, say, ABC News, Rasmussen, and Harris Interactive cannot be lumped together. Again, he should name names.
And then there is this key phrase: “Most of these surveys are not made public, but insiders can be made aware of them.” Ah, yes. Insiders. Of which I assume Charlie Cook is one. And that’s not an insult. I simply assume that campaign operatives, conscious of what Cook’s ratings might suggest about a race, would be happy to share private polls with him if they thought the poll results might move Cook’s ratings in a particular direction. I am not accusing Cook of anything at all; to me, his ratings have a lot of validity. I am merely making a statement about what campaign operatives do.
Which leads us to the obvious point, one well-known to any poll watcher and surely to Cook himself: the problem with private polls has nothing to do with methodology. It has to do with the incentives that govern whether those poll numbers are made public. For every “sophisticated,” “rigorous,” “reliable” private poll that is released, there are many others that never see the light of day because the results aren’t nearly so congenial to their sponsors. Private polls “shared with insiders” do not offer an unbiased account of any election. This “post”:http://www.huffingtonpost.com/2010/10/08/dueling-partisan-polls-co_n_755397.html by Mark Blumenthal cites some research to that effect.
I want to re-emphasize the main point in my “earlier post”:https://themonkeycage.org/2010/06/on_pollster_ratings.html on pollster ratings. Far too much ink is spilled judging polls by how well they predict election outcomes — a feat which most of them accomplish relatively well.
UPDATE: Tom Holbrook’s “post”:http://politics-by-the-numbers.blogspot.com/2010/10/are-polls-junk.html nicely depicts the accuracy of much pre-election polling.