Gallup’s Problems Are Everyone’s Problems

This is a magisterial exercise in polling forensics by Mark Blumenthal, with an assist from Brendan Nyhan.  I want to emphasize what Blumenthal notes at the end:

The real story here is less about Gallup than about the new reality of public opinion polling. Sophisticated random samples, live interviews and rigorous calling procedures alone can no longer guarantee accurate results. Today’s rapidly declining response rates require more weighting than ever before to correct demographic skews, a phenomenon that places growing stress on previously reliable weighting procedures.

In that context, the accepted standards for disclosure leave those who scrutinize polls in the dark. Survey firms routinely publish details such as polling dates, sample size and margin of error, but are less apt to share details about weighting. Many withhold the weighted demographic compositions of their adult samples, their “likely voter” samples or both, and few disclose their unweighted demographics or the severity of weighting required. Gallup releases the raw data for some of its polls—but only months after its original release of results and only to scholars at institutions that pay for access to the Roper Center archives. Gallup keeps most of the data for its massive daily tracking poll closely guarded.

If politicians, journalists and the general public want to understand why some pollsters’ results differ from others’, the standards for disclosure must change.

In an era when a poll is only as accurate as the procedures used to adjust its sample, and given the omnipresence of Gallup’s daily tracking poll in the coverage of campaign 2012, we need to know more.

This really can’t be said strongly enough.  I’m not someone with strong opinions about the methodology of most individual pollsters, including Gallup.  (I’m usually more frustrated with how poll results are presented.)  But the challenges facing the polling industry do demand transparency about methodology—to say nothing of data that is publicly available in a timely fashion—and many pollsters simply will not provide either.

2 Responses to Gallup’s Problems Are Everyone’s Problems

  1. RobC June 18, 2012 at 3:10 am #

    Even before one gets to descriptions of weighting methodology, isn’t there a more fundamental factual question of what any particular poll’s response rate is? Is there any justification for failure to report the response rate? If not, why isn’t it reported, other than that it raises uncomfortable questions about non-response bias?

    • Andrew Gelman June 18, 2012 at 7:28 am #

      I agree that it is a good idea to report response rate but it is not always so easy because you can never be sure what is the correct denominator. Suppose you call and call a phone number and never get a response. This could be a household with nobody home (so it should be in the denominator) or where people are at home but are not picking up the phone (so it should be in the denominator) or it could be a number not attached to any household (so it should not be in the denominator). See this article for some examples.

      Also, of course the resp rate can be very low but the poll numbers can be fine. But I agree that there’s no reason not to report the resp rate anyway.