This is a guest post by political scientist Sean Richey and Ben Taylor.
*****
On Election Day, we asked 565 Amazon Mechanical Turk (MTurk) workers to take a brief survey on vote choice, ideology and demographics. MTurk has three million online workers who do computer tasks for a small amount of money. For example, they could rate the likeability of an advertisement for a nickel. Because MTurk is easy, cheap, and fast, it is becoming a popular subject pool for experimental research. Our results cost $28 and came back to us in four hours from workers who live across the United States. (To sign up for free account, click here.)
Prior studies have found that MTurk workers respond similarly to representative samples of the United States on many political questions, and respond in similar ways to experimental stimuli (see this paper by Berinsky, Huber, and Lenz). These comparisons suggest that MTurk is a good sample for experiments, better than the college student samples that are commonplace. Although Berinsky, Huber, and Lenz find that MTurk workers are not fully representative, the ease of MTurk and the similarity of experimental results using MTurk to previous results make MTurk very attractive. More studies are needed, however, to investigate this resource.
We take a different approach here. We compare MTurk workers on Election Day to actual election results and exit polling. The survey paid $0.05 and had seven questions: gender, age, education, income, state of residence, vote choice, and ideology. Overall, 73% of these MTurk workers voted for Obama, 15% for Romney, and 12% for “Other.” This is skewed in expected ways, matching the stereotypical image of online IT workers as liberal — or possibly libertarian since 12% voted for a third party in 2012, compared to 1.6% percent of all voters.
Voter turnout was also skewed. Although turnout was around 60%, 86% of MTurk respondents reported voting. Some over-reporting is common in surveys, but rarely as much as here. Regardless of whether MTurk workers really do turn out more, or simply feel more pressure to say they did, they again differs substantially from the general population.
Ideology was also skewed. Almost 61% of these MTurk workers identified as liberals, but exit polls showed only 25% of voters on Election Day did.
Finally, the demographics of MTurk workers are also unrepresentative. Most importantly, the MTurk sample is heavily skewed toward younger people: 72% of the sample was 18 to 29 years old, compared to only 17% in the country as a whole. Those 65 and older are only 0.2% of our sample (actually only two respondents), while nationally they were 13%. Females are 50.8% of the United States, but only 34% of this sample. MTurk workers also have lower incomes, on average. About 57% made less than $50,000 each year (vs. 41% nationally), 31% made $50-100,000 (vs. 41%), and 11% made more than $100,000 (vs. 21%).
MTurk workers are also more highly educated. Few have no high school degree (2% MTurk vs. 14% Nationally) or only a high school degree (11% vs. 28%). There are far more with some college education (48% vs. 29%) or a college degree (29% vs. 18%).
In sum, the MTurk sample is younger, more male, poorer, and more highly educated than Americans generally. This matches the image of who you might think would be online doing computer tasks for a small amount of money.
The question of whether this sample is more diverse than a student sample depends on the school the students attend. At my university—Georgia State University in Atlanta, GA—this sample would be less diverse than our student body in many respects (except state of residence, of course). As the percentage of non-traditional students continues to rise, college student samples are becoming more diverse in terms of age, race and ethnicity, gender, income and so on.
We cannot speak to whether the experimental results derived from a representative sample could be replicated in this MTurk sample. However, the skew in vote choice, ideology, and demographics should be taken seriously when assessing the external validity of research using MTurk samples. MTurk is an exciting new method to find subjects for research, but its value deserves further study and exploration.