# Difficulty of detecting relative changes in opinion

Just to add to John’s criticism of silly publicity involving changes in the polls:

John mentions a report of “a March poll of young people that finds a 7-point difference between Obama and a generic Republican, a much smaller margin than between Obama and McCain in 2008” and points out that, before jumping to conclusions, journalists should “compare the trends among the group in focus to the trends among other groups. It doesn’t mean much if Obama is down among young voters if he’s down among middle-aged voters and seniors too. Often, swings among demographic groups are fairly uniform, which suggests that a candidate may not have a unique problem with one group but a systemic problem with many groups.”

I agree with John. We use the term “uniform partisan swing” to describe the above pattern, which appears in public opinion and elections. (See some examples here)

One thing I’d like to add to make John’s point even more forcefully is that any nonuniform swings are difficult to detect in a survey.

Here’s a quick calculation:

Suppose you have a random sample of 1600 Americans. The margin of error of the survey (that is, 2 standard errors) is 1/sqrt(1600) = 0.025; thus, you can estimate public opinion to within 2.5%.

Now consider a comparison, in which we compare 1/4 of the respondents to the other 3/4. The margin of error of the difference is sqrt (1/400 + 1/1200) = 0.057. So you can only reliably detect relative changes in opinion of at least 6 percentage points.

A 6 percentage point change, compared to the rest of the population. That’s a big change. Which brings us to John’s other point, that you need data from several surveys to find anything.

### 4 Responses to Difficulty of detecting relative changes in opinion

1. Foster April 21, 2012 at 10:34 am #

(“…silly publicity involving changes in the polls..”)

.

Huge subject. {media fixation, ignorance & abuse of political opinion polling}

Such polling (and phony microscopic opinion analysis) is perhaps of supposed value to professional political strategists — but nobody else.

Simply wait for the official poll results (formal public elections) to see what the population sample actually chooses.
Why is that so difficult ?

“Our present addiction to pollsters and forecasters is a symptom of our chronic uncertainty about the future…. We watch our experts read the entrails of statistical tables and graphs the way the ancients watched their soothsayers read the entrails of a chicken.” (–Eric Hoffer)

• Andrew Gelman April 21, 2012 at 2:28 pm #

Foster:

I’m interested in how different groups vote, and I’m interested in how opinions change during the campaign. I’m not only interested in geographic breakdowns of the vote (which is all you can learn from the election returns).

2. ADTS April 22, 2012 at 9:55 am #

Andrew Gelman:

I appreciate the exposition and certainly recognize your mastery of the topic. Moreover, I have taken both undergraduate and graduate work in statistics, (quantitative) methods, etc. At the same time, I’d very much like to go even more slowly via, say, an online textbook. (For example, why does one use two standard errors* and why does one arrive at what that number is by dividing one over the square root of the sample size?) Basically: is there a good online reference you’d recommend that I could use to walk myself through the basics?

*Similarly, is that (two standard errors) essentially the same as two standard deviations?

Thanks