Campaigns and elections

A Second Look at National Pollster Accuracy

Nov 7 '12

Here are two more takes at it.  First, courtesy of UNC Ph.D. student Brice Acree, takes my original plot and then adds an underlying measure of uncertainty — essentially, the margin of error for the estimated margin of victory.

Second, spurred on by a friend who is also a pollster, I calculated a different quantity — called “A” in the piece by Martin, Traugott, and Kennedy — that also captures predictive accuracy, is more robust to how polls treat undecided voters, and allows me to calculate a confidence interval.  (UPDATE: This quantity was first calculated in the analysis by Costas Panagopoulos that I mentioned in my first post.  However, I mistakenly overlooked this fact.  Thus, what I present here is an unintentional replication of what he did.)  Here, I included only polls that left the field on November 3 or later, in an effort to exclude polls who polled earlier and may simply have missed late movement in the polls (as opposed to being inaccurate per se).  I averaged the two PPP polls during this period.

Although A has no intuitive metric, the results are sensible.  Rasmussen and Gallup, for example, had a more pro-Romney bias.  (“Bias” here is a statistical term of art and does not imply any partisan agenda on their part.)  PPP, YouGov, and Ipsos/Reuters had almost no bias.   Other polls, like ABC/Washington Post and NBC/WSJ had very minimal bias.

But note that because we are looking mostly at individual polls without very large samples, the underlying uncertainty is large.  [UPDATE: Note that this is what Panagopoulos observed as well — “but none of the 28 national pre-election polls I examined had a significant partisan bias.”]