Fourteen magic words: an update

In the discussion of the fourteen magic words that can increase voter turnout by over 10 percentage points, questions were raised about the methods used to estimate the experimental effects. I sent these on to Chris Bryan, the author of the study, and he gave the following response:

We’re happy to address the questions that have come up. It’s always noteworthy when a precise psychological manipulation like this one generates a large effect on a meaningful outcome. Such findings illustrate the power of the underlying psychological process.

I’ve provided the contingency tables for the two turnout experiments below. As indicated in the paper, the data are analyzed using logistic regressions. The change in chi-squared statistic represents the significance of the noun vs. verb condition variable in predicting turnout; that is, the change in the model’s significance when the condition variable is added. This is a standard way to analyze dichotomous outcomes.

Four outliers were excluded in Experiment 2. These were identified following a procedure described by Tabachnick and Fidell (5th Ed., pp. 443, 446-7) using standardized residuals; the z = 2.58 criterion was based on G. David Garson’s online publication “Statnotes: Topics in Multivariate analysis.” We think excluding these values is the appropriate way to analyze these data; if they are retained, the difference between conditions reduces to 9.4 percentage points (still a considerable difference), and the P-value increases to just under 0.15. Regardless, the larger point is that the effect replicated in the second, larger and more representative study (Experiment 3) where, incidentally, no outliers were excluded.

We agree that it is important to test the effects of this manipulation with larger samples; doing so would address the applied implications of the study—can this technique be used to increase turnout on a population scale and by how much? Nonetheless, as compared to typical psychology studies, the sample sizes are ample and, as the effects are consistently statistically significant, they clearly demonstrate the important psychological process we were interested in: that subtle linguistic cues that evoke the self can motivate socially-desirable behavior like voting.

I agree that the timing of the exercise—completed the day before and the morning of Election Day—was likely important, although the degree to which the effect decays over time is an important topic for future research. It’s also relevant that we manipulated the phrasing of 10 survey questions, not just one. So, while the difference between conditions was subtle, participants were exposed to it multiple times.

I hope this response is helpful. I’d be happy to address any further questions. In addition, if anyone is interested in collaborating on a larger-scale implementation of this experiment, I [Chris Bryan] would be excited to talk about that.

Contingency tables

Experiment 2:
Noun: 42 (voted), 2(did not)
Verb: 36 (voted), 8 (did not)

Experiment 3
Noun: 98 (voted), 11 (did not)
Verb: 83 (voted), 22 (did not)

Interesting and important for psychology (the idea that ideas of essentialism affect important real-world decisions) and for political science (the idea of political participation being connected to one’s view of oneself).

For my taste, the statistical analysis is way too focused on p-values and hypothesis testing—-I’m not particularly interested in testing hypotheses of zero effect, as I think everything has some effect, the real question being how large it is—-but, what can I say, that’s how they do things in psychology research (psychometrics aside). I’m guessing that the 10 percentage points is an overestimate of the effect. Also, I don’t quite understand the bit about outliers: if the outcomes are simply yes or no, what does it mean to be an outlier?

In any case, I think it’s great to have such discussions out in the open. This is the way to move forward.

4 Responses to Fourteen magic words: an update

  1. Dustin Cho September 10, 2011 at 12:51 pm #

    Very interesting. I look forward to larger scale replications for more precise estimates of the size of the effect.

    One thing that surprises me is the overall turnout rates — 89.9% and 79% in Experiment 3. As the authors point out, this is much, much higher than the 47% statewide turnout among registered voters. Apparently, participants were “invited to participate” by Knowledge Networks, a consumer survey firm. These incredibly high turnout rates make me concerned about the generalizability of the study. It sounds like the treatments weren’t administered much like professional get-out-the-vote campaigns.

    And as someone who conducted one of several studies attempting and failing to replicate Greenwald et al.’s shocking 1987 finding that merely asking someone whether they expect to vote increases turnout by 25%, I hope the replication studies get the attention they deserve! (In Nudge, Sunstein and Thaler apparently weren’t aware of the replication attempts when they cited Greenwald 1987 and wrote, “It turns out that if you ask people, the day before the election, whether they intend to vote, you can increase the probability of their voting by as much as 25 percent!”)

    • Andrew Gelman September 10, 2011 at 4:44 pm #


      Sunstein seems to have a pattern of accepting quantitative research claims at face value. Another time he expressed support for the death penalty on utilitarian grounds, based on the study that purported to show that every execution saves 18 lives.

  2. Mark B. September 10, 2011 at 1:00 pm #

    Regarding the “outliers”: If I’m interpreting the paper and the above clarification correctly, they used demographic data to predict peoples’ probability of voting . If the actual voting choice was significantly different from the predicted vote based on demographics, then I believe you could get a large standardized residual.

    My guess is that the “outliers” were people who, based on demographics, were very much likely to vote but then did not vote.

  3. Anonymous methodologist September 11, 2011 at 10:28 am #

    Chris, thanks for the response, and Andrew, thanks for taking the initiative to keep up the conversation.

    The large estimate found in the paper may indicate that this has some effect but it seems to me this must (essentially mechanically) be a large overestimate.

    Take a look at experiment 3. If we take the estimates seriously then if both groups were administered the verb condition, 21% of participants would not have voted. The claim in the paper is that half of this group (50%!) was induced by subtle wording on a questionnaire to go and vote — against all the other reasons they were not originally planning to vote. If we assume some part of the population would not vote no matter what (e.g. they are sick or hospitalized, traveling, etc), say 5%, then this might mean you move about 66% of otherwise non-voters to go to the polls!

    Is there some hole in my logic here? Don’t others find this a bit incredible too?

    If we agree on that then we could go back to the methodology and think through how this should be analyzed and how the publishing process should work for findings such as these.