In the discussion of the fourteen magic words that can increase voter turnout by over 10 percentage points, questions were raised about the methods used to estimate the experimental effects. I sent these on to Chris Bryan, the author of the study, and he gave the following response:
We’re happy to address the questions that have come up. It’s always noteworthy when a precise psychological manipulation like this one generates a large effect on a meaningful outcome. Such findings illustrate the power of the underlying psychological process.
I’ve provided the contingency tables for the two turnout experiments below. As indicated in the paper, the data are analyzed using logistic regressions. The change in chi-squared statistic represents the significance of the noun vs. verb condition variable in predicting turnout; that is, the change in the model’s significance when the condition variable is added. This is a standard way to analyze dichotomous outcomes.
Four outliers were excluded in Experiment 2. These were identified following a procedure described by Tabachnick and Fidell (5th Ed., pp. 443, 446-7) using standardized residuals; the z = 2.58 criterion was based on G. David Garson’s online publication “Statnotes: Topics in Multivariate analysis.” We think excluding these values is the appropriate way to analyze these data; if they are retained, the difference between conditions reduces to 9.4 percentage points (still a considerable difference), and the P-value increases to just under 0.15. Regardless, the larger point is that the effect replicated in the second, larger and more representative study (Experiment 3) where, incidentally, no outliers were excluded.
We agree that it is important to test the effects of this manipulation with larger samples; doing so would address the applied implications of the study–can this technique be used to increase turnout on a population scale and by how much? Nonetheless, as compared to typical psychology studies, the sample sizes are ample and, as the effects are consistently statistically significant, they clearly demonstrate the important psychological process we were interested in: that subtle linguistic cues that evoke the self can motivate socially-desirable behavior like voting.
I agree that the timing of the exercise–completed the day before and the morning of Election Day–was likely important, although the degree to which the effect decays over time is an important topic for future research. It’s also relevant that we manipulated the phrasing of 10 survey questions, not just one. So, while the difference between conditions was subtle, participants were exposed to it multiple times.
I hope this response is helpful. I’d be happy to address any further questions. In addition, if anyone is interested in collaborating on a larger-scale implementation of this experiment, I [Chris Bryan] would be excited to talk about that.
Noun: 42 (voted), 2(did not)
Verb: 36 (voted), 8 (did not)
Noun: 98 (voted), 11 (did not)
Verb: 83 (voted), 22 (did not)
Interesting and important for psychology (the idea that ideas of essentialism affect important real-world decisions) and for political science (the idea of political participation being connected to one’s view of oneself).
For my taste, the statistical analysis is way too focused on p-values and hypothesis testing—I’m not particularly interested in testing hypotheses of zero effect, as I think everything has some effect, the real question being how large it is—but, what can I say, that’s how they do things in psychology research (psychometrics aside). I’m guessing that the 10 percentage points is an overestimate of the effect. Also, I don’t quite understand the bit about outliers: if the outcomes are simply yes or no, what does it mean to be an outlier?
In any case, I think it’s great to have such discussions out in the open. This is the way to move forward.