Difficulty of detecting relative changes in opinion

Just to add to John’s criticism of silly publicity involving changes in the polls:

John mentions a report of “a March poll of young people that finds a 7-point difference between Obama and a generic Republican, a much smaller margin than between Obama and McCain in 2008” and points out that, before jumping to conclusions, journalists should “compare the trends among the group in focus to the trends among other groups. It doesn’t mean much if Obama is down among young voters if he’s down among middle-aged voters and seniors too. Often, swings among demographic groups are fairly uniform, which suggests that a candidate may not have a unique problem with one group but a systemic problem with many groups.”

I agree with John. We use the term “uniform partisan swing” to describe the above pattern, which appears in public opinion and elections. (See some examples here)

One thing I’d like to add to make John’s point even more forcefully is that any nonuniform swings are difficult to detect in a survey.

Here’s a quick calculation:

Suppose you have a random sample of 1600 Americans. The margin of error of the survey (that is, 2 standard errors) is 1/sqrt(1600) = 0.025; thus, you can estimate public opinion to within 2.5%.

Now consider a comparison, in which we compare 1/4 of the respondents to the other 3/4. The margin of error of the difference is sqrt (1/400 + 1/1200) = 0.057. So you can only reliably detect relative changes in opinion of at least 6 percentage points.

A 6 percentage point change, compared to the rest of the population. That’s a big change. Which brings us to John’s other point, that you need data from several surveys to find anything.

4 Responses to Difficulty of detecting relative changes in opinion

  1. Foster April 21, 2012 at 10:34 am #

    (“…silly publicity involving changes in the polls..”)


    Huge subject. {media fixation, ignorance & abuse of political opinion polling}

    Such polling (and phony microscopic opinion analysis) is perhaps of supposed value to professional political strategists — but nobody else.

    Simply wait for the official poll results (formal public elections) to see what the population sample actually chooses.
    Why is that so difficult ?

    “Our present addiction to pollsters and forecasters is a symptom of our chronic uncertainty about the future…. We watch our experts read the entrails of statistical tables and graphs the way the ancients watched their soothsayers read the entrails of a chicken.” (–Eric Hoffer)

    • Andrew Gelman April 21, 2012 at 2:28 pm #


      I’m interested in how different groups vote, and I’m interested in how opinions change during the campaign. I’m not only interested in geographic breakdowns of the vote (which is all you can learn from the election returns).

  2. ADTS April 22, 2012 at 9:55 am #

    Andrew Gelman:

    I appreciate the exposition and certainly recognize your mastery of the topic. Moreover, I have taken both undergraduate and graduate work in statistics, (quantitative) methods, etc. At the same time, I’d very much like to go even more slowly via, say, an online textbook. (For example, why does one use two standard errors* and why does one arrive at what that number is by dividing one over the square root of the sample size?) Basically: is there a good online reference you’d recommend that I could use to walk myself through the basics?

    *Similarly, is that (two standard errors) essentially the same as two standard deviations?


  3. Nadia April 22, 2012 at 8:55 pm #

    To the two of you, how do you recommend interpreting internals in surveys and evaluating dynamics and messaging? Ezra wrote a terrific take down in the Washington Post, but some messages do seem to come through as well.

    For instance, the word “rich” came up much more frequently in open-ended descriptions of Romney in the spring compared to the fall. And people showed a strong propensity to think that Romney’s policies would favor the rich. This also seemed to show up in questions that Diane Sawyer got when she asked viewers for questions for her Romney interview. In a focus group of Republicans and Republican leaning independents, voters said brought up this about Romney. Of course, Professor Sides noted on fivethirtyeight that this need not be exaggerated. But, nonetheless this idea of Romney did seem to sink into the public’s view.

    Prof. Gelman, you noted that the conflicting opinions on inequality represented an opportunity for smart politicians to shape public opinion in their direction. The latest WSJ/NBC poll seemed to bear out that point, with the pollster’s testing messages and finding some of Obama’s doing better than any of Romney’s.

    Several surveys have shown an advantage for Romney on the economy per se, and some interpret that to say that Romney can win by talking about the economy. Lynn Vavreck admires the Obama isn’t working website but actually thinks it’s not a terribly effective message for the Romney team. She and Thad Kousser noted the weirdness of a counterfactual message. That seemed to be borne out by certain episodes like Romney visiting a factory in Ohio that closed before Obama even took office, and this claim that 90% of job losses were female in the Obama years, which independent fact checkers skewered as being misleading because of the cycles of a recession. On the one hand, as Ezra Klein points out, voters don’t pay close attention to this kind of thing. But these episodes don’t seem to be emblematic of successful “clarifying campaign” messaging.