Campaigns and elections

Difficulty of detecting relative changes in opinion

Apr 20 '12

Just to add to John’s criticism of silly publicity involving changes in the polls:

John mentions a report of “a March poll of young people that finds a 7-point difference between Obama and a generic Republican, a much smaller margin than between Obama and McCain in 2008” and points out that, before jumping to conclusions, journalists should “compare the trends among the group in focus to the trends among other groups. It doesn’t mean much if Obama is down among young voters if he’s down among middle-aged voters and seniors too. Often, swings among demographic groups are fairly uniform, which suggests that a candidate may not have a unique problem with one group but a systemic problem with many groups.”

I agree with John. We use the term “uniform partisan swing” to describe the above pattern, which appears in public opinion and elections. (See some examples here)

One thing I’d like to add to make John’s point even more forcefully is that any nonuniform swings are difficult to detect in a survey.

Here’s a quick calculation:

Suppose you have a random sample of 1600 Americans. The margin of error of the survey (that is, 2 standard errors) is 1/sqrt(1600) = 0.025; thus, you can estimate public opinion to within 2.5%.

Now consider a comparison, in which we compare 1/4 of the respondents to the other 3/4. The margin of error of the difference is sqrt (1/400 + 1/1200) = 0.057. So you can only reliably detect relative changes in opinion of at least 6 percentage points.

A 6 percentage point change, compared to the rest of the population. That’s a big change. Which brings us to John’s other point, that you need data from several surveys to find anything.