Fourteen magic words that can increase voter turnout by over 10 percentage points??

by Andrew Gelman on September 4, 2011 · 14 comments

in Campaigns and elections

Christopher Bryan, Gregory Walton, Todd Rogers, and Carol Dweck did two experiments in which they increased people’s voter turnout in real electionsby over 10 percentage points by simply asking them the following survey question on election day:

How important is it to you to be a voter in the upcoming election?

In the comparison condition, potential voters were asked:

How important is it to you to vote in the upcoming election?

Unfortunately, there doesn’t seem to have been a control condition in the experiments, so all they could really do was compare these two treatments to each other.

The gimmick of the experiment is that it harnesses humans’ natural belief in essentialism (see, for example, reference 14 in the link above), the idea that being “a voter” is more essential than being a person who happened to vote.

As Bryan et al. put it, “people may be more likely to vote when voting is represented as an expression of self—as symbolic of a person’s fundamental character—rather than as simply a behavior.”

This all makes sense. What about the estimated effect size of 10-14 percentage points? That sounds implausibly huge: given all the campaigning that goes on, could a single survey question really have that large an effect? I can’t believe it. Well, maybe. The key is that the experiment was done on election day itself (or the day immediately before). I suppose if somebody catches you just at the right time, it can make a big difference.

{ 14 comments }

Hans Noel September 4, 2011 at 1:14 pm

Interesting. It suggests a new question. How closely do people’s self-expressions match their behavior? Can one think of oneself as a voter who happened not to vote this time? (I know I’ve thought of myself as a nonsmoker my entire life, including the period in college when I sometimes smoked cigars and cigarettes. )

John Transue September 4, 2011 at 3:54 pm

This is interesting to me because I’ve seen mixed results with similar treatments. We had a POQ on this topic that strikes me as congruent with this paper:
Diana Burgess, Beth Haney, Mark Snyder, John L. Sullivan, and John E. Transue “Rocking the Vote: Using Personalized Messages to Motivate Voting among Young Adults.” Public Opin Q (2000) 64(1): 29-52 doi:10.1086/316758

When Rock the Vote was registering people, they had been also creating and storing reminder postcards, which they then mailed to the people they had registered just before the election. During the registration campaign, they had to print up a new batch and somebody (I think Ricki Seidman) decided to add a prompt that said, “I will vote because ____,” and they could fill in whatever they wanted. We found that receiving the postcard with the prompt increased turnout.

On the other hand, another group did something very similar over the phone and found no effect:
Smith, J. K., Gerber, A. S. and Orlich, A. (2003), Self-Prophecy Effects and Voter Turnout: An Experimental Replication. Political Psychology, 24: 593–604. doi: 10.1111/0162-895X.00342

“Eligible respondents were contacted by tele- phone on the Sunday and Monday evenings immediately preceding the 7 March 2000 presidential primary,” (page 595).

They had several versions of a self-prophecy treatment:
“The interview for respondents assigned to the compound treatment proceeded as follows:
What do you expect to do between now and the time the polls close tomorrow [or Tuesday]? Do you expect that you will vote, or not?

Following the Greenwald protocol, respondents who replied that they didn’t know whether they would vote were pressed to respond one way or the other:
We would like you to predict your action in any case. Do you think you will vote or not?

Callers completed the interview by asking:
What would you say is the most important single reason for voting? ” (page 597).

To the extent that the Rock the Vote prompt is more of a self-expression than a prediction and justification of a person’s behavior, this article might explain why the results were so different across the two studies mentioned above.

Andrew Gelman September 4, 2011 at 9:12 pm

A little cognitive psychology goes a long way, apparently.

doug September 5, 2011 at 11:12 am

10% effect? Personally, I find Bem’s 3% ESP effect to be more plausible.

Anonymous methodologist September 5, 2011 at 12:44 pm

I find this paper very hard to believe.

First, the very small sample sizes (especially for this kind of experiment) should be a reason to worry that this is just by chance. And the methods of analysis are questionable, I think. Andrew, wouldn’t the right test here be Fisher’s exact test, rather than to rely on chi-square distributional assumptions?

The authors don’t give the actual numbers (which they of course should) but from the percentages it looks like we have two contingency tables (from experiment 2 and 3):

TX CT
Voted 43 36
Not 2 8

and
TX CT
Voted 96 85
Not 11 22

If you use Fisher’s exact test on these tables you get a P = 0.050 and P=0.057 for experiments 2 and 3. If, in experiment 2 you had 3 people not vote in the treatment condition, instead of 2, the p-value would be P = 0.11. No matter what the right test is, these small sample sizes leave a lot of room for chance.

I also have a hard time seeing the justification for eliminating the outliers in experiment 2 (and for not showing tests including these “outliers”). After all, they are not outliers in the conventional sense (as a value of a continuous variable way out of range) but rather a binary response that doesn’t fit the authors model.

Andrew Gelman September 5, 2011 at 1:16 pm

The appropriate analysis is neither the Fisher test or a chi-squared test. I would use a logistic regression. For some reason, psychologists like to do hypothesis tests but I find that whole approach a bit indirect.

Dan Munz September 5, 2011 at 4:10 pm

I’ve long suspected an effect like this is behind some people’s personal solutions to the “irrationality of voting” problem. People identifying with a group makes it more plausible that their vote is consequential. (i.e., “I” can’t swing the election, but “Latinos” can swing the election, and I’m a Latino, so I’d better go vote.) This isn’t the same effect, but the role of group identification in overcoming the irrationality of voting seems to me like fertile ground.

Andrew Gelman September 5, 2011 at 4:38 pm

Dan:

Yes, people have studied this. Carole Uhlaner wrote a paper in the late 80s, I’m sure there’s been more done on this too.

Dan Munz September 5, 2011 at 7:39 pm
Anonymous methodologist September 5, 2011 at 4:26 pm

Andrew, thanks for the response. I appreciate that the Fisher’s test can be a bit conservative but that at least should be strictly preferred to a chi-squared test with only 2 or 3 observations in some cells (at least one in this case).

I ran a quick logistic, with robust standard errors, and using the contingency tables above I get a P = 0.067 for experiment 2 and P = 0.047 for experiment 3.

These are decidedly more marginal than given in the paper (of course I don’t know the exact contingency tables, but these should be close). They also hide the uncertainty that comes with the low sample size. For example, if the 4 cases deleted from the analysis of experiment 2 are assumed to be two from each group and happen to go against the hypothesis, i.e. if the two treatment cases didn’t vote and the 2 control cases did, then the P-value for experiment 2 would jump to P = 0.21.

These types of experiments are regularly conducted with thousands of voters, and are relatively inexpensive, so I would have liked to see the authors replicate this with a larger sample before submitting for publication.

Andrew Gelman September 5, 2011 at 4:42 pm

I’ll ask Rogers what he thinks and I’ll get back to you with any response.

Anonymous methodologist September 5, 2011 at 5:14 pm

Sounds good. It would be good to at least get the actual contingency tables so that we can evaluate the different methods of analysis using the correct data.

Also, this is of course just a quick analysis on my end, and it would be good for the discussion if somebody else could replicate the logistic regressions to make sure I’m running this all correctly.

Andrew Gelman September 10, 2011 at 11:35 am

See the authors’ response to your questions here.

Austin Paulnack September 8, 2011 at 8:26 pm

After years of working on voter registration in Syracuse NY, particularly at Syracuse University, I have found that the most important factor in increasing voting from new voters is simply to confront them with a voter reg form (in dining halls, front of Starbucks, at campus bus stops), help them fill it out, and take it back and deliver it to the Board of Elections. And follow up, with a reminder in the days before the November election, to get out the vote at their nearby polling place. At Syracuse University, fraternities, sororities, et. al. found that getting involved in voter registration was an easy way to meet many members of the opposite sex. Austin Paulnack

Comments on this entry are closed.

Previous post:

Next post: