Obama up in Arizona? Romney leading big in Florida? The frenzied state of contemporary poll watching seems to be something close to this classic XKCD comic. In that context, it is worth repeating a point that John Sides has made here in recent days, and that Gary King has made elsewhere: we should look at polling data for broad trends, not individual results.
Still, analysts do need to be on the lookout for systematic sources of bias that are likely to influence many polls, and so to throw off aggregated estimates. On average, publicly released polls have been performing well in recent years. But with declining response rates, increasing numbers of cellphone-only households, and more Spanish-speaking voters, that performance is not something we should take for granted.
Now, the bias induced by omitting cellphone-only households is well known. But less appreciated is the fact that the cellphone bias can grow when we try to survey geographically targeted areas, like, I don’t know, Virginia. Without getting actual location data from telephone companies or sampling cellphones nationwide, pollsters are forced to ignore people who move into a given state with an out-of-state cell number. Consider the manifestly unrepresentative sample that resides in my cellphone. Of the roughly 100 cellphone numbers I have saved, 49% are owned by people living (and presumably voting) in states that do not match their cellphone’s area code. More systematically, Pew Research notes that approximately 10 percent of cellphone respondents live in a different state than their area code implies. Even a pollster willing to pay a premium to call cellphones would miss these people in a geographically targeted survey. (And the problem is worse still when polling in city-level or Congressional races, since an in-state move can also cause problems in those cases.) I don’t know of a single state-level polling firm that typically includes these out-of-state cellphones in publicly released polls. Quinnipiac, one oft-cited pollster of U.S. states, does not.
Beyond my friends and family, is this likely to matter? Consider the American Community Survey, which provides data on year-to-year internal migration. The 2011 data indicate that 5.2% of all 18 to 24 year-olds and 4.0% of all 25 to 34 year-olds moved across state lines in the prior year, as compared to a baseline rate of 2.3% for Americans over 1. Moving across state boundaries is also more common at higher levels of educational attainment, with 1.5% of high school graduates but 3.0% of those with graduate or professional degrees having moved across state lines in the last year. Some of those movers get new cellphone numbers, of course, but many others don’t. (From my foray into my own cellphone’s memory, I’d venture the following hypothesis. The stronger your attachment to your former area code, and the cooler that area code is thought to be, the less likely you are to switch. I call people in New York City a lot less frequently than my phone bill implies.)
Using 2012 Pew data, I’ve simulated the likely partisan bias resulting from under-sampling these demographic groups, and it doesn’t have a strong impact on the partisan composition of the sample. But data limitations mean that I don’t directly observe whether people’s cellphones are out-of-state–and have to rely on some strong assumptions instead. It is also quite plausible that the bias in missing more mobile people with lots of education is offset by the positive relationship between education levels and responsiveness to surveys generally. Still, it’s a bias to keep an eye on. That is especially true at a time when age is increasingly correlated with partisan preferences. And it is especially true for states like Colorado, Nevada, New Hampshire, and Virginia, all of which have seen above-average levels of domestic migration in recent years.