Remember that “Daily Kos/Research 2000 Poll”:http://www.dailykos.com/storyonly/2010/2/2/832988/-The-2010-Comprehensive-Daily-Kos-Research-2000-Poll-of-Self-Identified-Republicans from earlier this week that swept through the blogosphere (see “here”:http://www.politico.com/news/stories/0210/32384.html, “here”:http://voices.washingtonpost.com/ezra-klein/2010/02/what_republicans_believe_about.html , and “here”:http://www.conservatives4palin.com/2010/02/dailykosresearch-2000-poll-palin-leads.html). Well, like any poll with new, dramatic take-away healdines, it’s probably worth a second look. I had put that on my Monkey Cage to do list for later this week, only to discover one of my own graduate students, “Andrew Therriault”:http://andrewtherriault.com/ (who incidentally will be on the job market next year), had already beaten me to the punch, providing a – dare I say it – fair and balanced look at the poll’s methodology “here”:http://andrewtherriault.com/index.php/2010/02/03/daily-kos-research-2000-poll/, which he has graciously allowed me to re-post on the Monkey Cage as a guest post:
There’s a new poll out from Daily Kos, conducted by Research 2000 (“story”:http://www.dailykos.com/storyonly/2010/2/2/832988/-The-2010-Comprehensive-Daily-Kos-Research-2000-Poll-of-Self-Identified-Republicans, “crosstabs”:http://www.dailykos.com/statepoll/2010/1/31/US/437), that’s getting a lot of attention this week (see discussion at “FiveThirtyEight”:http://www.fivethirtyeight.com/2010/02/mcgop-virtues-and-vices-of-sameness.html and “Politico”:href=http://www.politico.com/news/stories/0210/32431.html, for example). In brief, it claims that an alarming number of Republicans believe that Obama wants the terrorists to win, believe that ACORN stole the 2008 election, and hold other similarly-extreme beliefs and opinions. While the findings are pretty striking at first glance, there are a number of potential problems with the poll that should throw a little cold water on anyone getting too hysterical about the results:
Sample selection
The poll asked these questions of “2003 self identified Republicans”, but no details are provided about the screening process—what the specific eligibility criteria were, what the response rate was, what percentage of respondents fit the eligibility screen, and so forth. I would wager that Republican leaners are not included, but that’s only part of the issue. The poll measures the opinions of people who (a) answered the phone and were willing to be polled far from election day, (b) identified as Republican without any follow-up prompts, c) were interested and patient enough to sit through a moderately-lengthy survey, and (d) did this despite a list of questions which sounds awfully like a “push poll”:http://en.wikipedia.org/wiki/Push_poll. Each of these factors could be reasonably expected to favor respondents who are highly engaged with politics and predisposed toward a particularly conservative viewpoint. As such, it is highly unlikely that the sample of respondents who sat through the full survey is even close to representative of the typical Republican electorate.
Opinion strength
Every opinion question is binary (yes/no, favor/oppose, etc.) with an option for “not sure”. Looking at the percentage of “not sure” responses, almost every question has double-digits in this category, and many have 20-30% or more. This is a much greater incidence than for most survey questions (though data is scarce when it comes to questions comparable to these in tone), and suggests that there is a wide range when it comes to the strength and certainty of respondents’ opinions. So of the 63% who think Obama is a socialist, for example, it’s unlikely that all of those respondents think he’s the reincarnation of V.I. Lenin. More likely, a handful really believe that, some more think he’s socialist in the European, democratic-socialist sense, others have heard their friends say it and think it might be true, a few more don’t really know but are guessing (not wanting to admit to the interviewer that they don’t know), and a bunch have no idea what a socialist is in the first place but know that it’s evil and so Obama must be one. By only allowing for binary answers, this poll ignores the complexities and uncertainties of public opinion, and force responses into categories which sound much more extreme than they might otherwise be.
Remember that “Daily Kos/Research 2000 Poll”:http://www.dailykos.com/storyonly/2010/2/2/832988/-The-2010-Comprehensive-Daily-Kos-Research-2000-Poll-of-Self-Identified-Republicans from earlier this week that swept through the blogosphere (see “here”:http://www.politico.com/news/stories/0210/32384.html, “here”:http://voices.washingtonpost.com/ezra-klein/2010/02/what_republicans_believe_about.html, and here). Well, like any poll with new, dramatic take-away healdines, it’s probably worth a second look. I had put that on my Monkey Cage to do list for later this week, only to discover one of my own graduate students, “Andrew Therriault”:http://andrewtherriault.com/ (who incidentally will be on the job market next year), had already beaten me to the punch, providing a – dare I say it – fair and balanced look at the poll’s methodology “here”:http://andrewtherriault.com/index.php/2010/02/03/daily-kos-research-2000-poll/, which he has graciously allowed me to re-post on the Monkey Cage as a guest post:
There’s a new poll out from Daily Kos, conducted by Research 2000 (“story”:http://www.dailykos.com/storyonly/2010/2/2/832988/-The-2010-Comprehensive-Daily-Kos-Research-2000-Poll-of-Self-Identified-Republicans, “crosstabs”:http://www.dailykos.com/statepoll/2010/1/31/US/437), that’s getting a lot of attention this week (see discussion at “FiveThirtyEight”:http://www.fivethirtyeight.com/2010/02/mcgop-virtues-and-vices-of-sameness.html and “Politico”:href=http://www.politico.com/news/stories/0210/32431.html, for example). In brief, it claims that an alarming number of Republicans believe that Obama wants the terrorists to win, believe that ACORN stole the 2008 election, and hold other similarly-extreme beliefs and opinions. While the findings are pretty striking at first glance, there are a number of potential problems with the poll that should throw a little cold water on anyone getting too hysterical about the results:
<strong>Sample selection</strong>
The poll asked these questions of “2003 self identified Republicans”, but no details are provided about the screening process—what the specific eligibility criteria were, what the response rate was, what percentage of respondents fit the eligibility screen, and so forth. I would wager that Republican leaners are not included, but that’s only part of the issue. The poll measures the opinions of people who (a) answered the phone and were willing to be polled far from election day, (b) identified as Republican without any follow-up prompts, c) were interested and patient enough to sit through a moderately-lengthy survey, and (d) did this despite a list of questions which sounds awfully like a “push poll”:http://en.wikipedia.org/wiki/Push_poll. Each of these factors could be reasonably expected to favor respondents who are highly engaged with politics and predisposed toward a particularly conservative viewpoint. As such, it is highly unlikely that the sample of respondents who sat through the full survey is even close to representative of the typical Republican electorate.
<strong>Opinion strength</strong>
Every opinion question is binary (yes/no, favor/oppose, etc.) with an option for “not sure”. Looking at the percentage of “not sure” responses, almost every question has double-digits in this category, and many have 20-30% or more. This is a much greater incidence than for most survey questions (though data is scarce when it comes to questions comparable to these in tone), and suggests that there is a wide range when it comes to the strength and certainty of respondents’ opinions. So of the 63% who think Obama is a socialist, for example, it’s unlikely that all of those respondents think he’s the reincarnation of V.I. Lenin. More likely, a handful really believe that, some more think he’s socialist in the European, democratic-socialist sense, others have heard their friends say it and think it might be true, a few more don’t really know but are guessing (not wanting to admit to the interviewer that they don’t know), and a bunch have no idea what a socialist is in the first place but know that it’s evil and so Obama must be one. By only allowing for binary answers, this poll ignores the complexities and uncertainties of public opinion, and force responses into categories which sound much more extreme than they might otherwise be.
Consistency bias
There is a common theme in nearly all these questions: they ask respondents whether they agree with viewpoints espoused by the most far-right commentators. In most every case, there is a clear “conservative” position to be taken. This could lead to more extreme results than otherwise for two reasons. First, a respondent may be inclined to give consistent answers from one question to the next, choosing the conservative position as a default and only deviating for questions which are beyond the pale even by far-right standards (e.g., whether women should be allowed to work). And second, reminding respondents of all the far-right accusations and opinions may serve to activate the respondents’ more extreme attitudes—a respondent who might be unsure, for example, whether Sarah Palin is more qualified than Barack Obama would probably be more inclined to favor Palin after hearing the suggestion that Obama may be an impeachment-worthy, non-citizen socialist who wants the terrorists to win than she would be independent of these prompts.
House effects
Finally, note that Daily Kos commissioned this poll from Research 2000 in order to provide material for his upcoming book on the extreme beliefs of the far-right. Daily Kos and Research 2000 have a long partnership, and their results have shown a fairly serious house effect; in 2008, for example, Pollster rated their presidential tracking poll as having the “largest house effects”:http://www.pollster.com/blogs/tracking_poll_house_effects.php of all the major tracking polls. Perhaps not surprisingly, this effect is generally in the liberal direction. This is not to in any way imply that Research 2000 is cooking the numbers—each polling organization uses its own unique methods for collecting and analyzing data, so results will vary accordingly—but we should remember that there is a business side of this and be realistic about the incentives in play. In this case, Kos’s interest is in data which shows the far-right to be really far-right, and Research 2000 likely wants to keep one of its biggest clients happy. It’s not hard to see, then, how the methodological choices made (consciously or unconsciously) when designing this poll could make extreme results more likely.
Many of these concerns could be alleviated (or substantiated) if more data (information on the screening process, response/completion rates, and individual-level responses) were made available. In the meantime, however, I think we need to be sober and skeptical about the results of this poll. There are without a doubt a lot of crazies on the right (as there are on the left), but whether they make up the majority of the Republican party is another question altogether. These results are certainly interesting and perhaps alarming, but they are far from conclusive.