This is a magisterial exercise in polling forensics by Mark Blumenthal, with an assist from Brendan Nyhan. I want to emphasize what Blumenthal notes at the end:
The real story here is less about Gallup than about the new reality of public opinion polling. Sophisticated random samples, live interviews and rigorous calling procedures alone can no longer guarantee accurate results. Today’s rapidly declining response rates require more weighting than ever before to correct demographic skews, a phenomenon that places growing stress on previously reliable weighting procedures.
In that context, the accepted standards for disclosure leave those who scrutinize polls in the dark. Survey firms routinely publish details such as polling dates, sample size and margin of error, but are less apt to share details about weighting. Many withhold the weighted demographic compositions of their adult samples, their “likely voter” samples or both, and few disclose their unweighted demographics or the severity of weighting required. Gallup releases the raw data for some of its polls—but only months after its original release of results and only to scholars at institutions that pay for access to the Roper Center archives. Gallup keeps most of the data for its massive daily tracking poll closely guarded.
If politicians, journalists and the general public want to understand why some pollsters’ results differ from others’, the standards for disclosure must change.
In an era when a poll is only as accurate as the procedures used to adjust its sample, and given the omnipresence of Gallup’s daily tracking poll in the coverage of campaign 2012, we need to know more.
This really can’t be said strongly enough. I’m not someone with strong opinions about the methodology of most individual pollsters, including Gallup. (I’m usually more frustrated with how poll results are presented.) But the challenges facing the polling industry do demand transparency about methodology—to say nothing of data that is publicly available in a timely fashion—and many pollsters simply will not provide either.