Request for Feedback on Experiment: Corruption and Voting

As anyone who involved in survey research can tell you, one of the more frustrating comments you can get at talks is when people tell you that you should have asked one of your questions differently. While often the questioner has a good point, the fact that you’ve already run the survey introduces a minor time-inconsistency problem: the best time to get this kind of feedback is before the survey goes into the field. However, most of the time when someone wants you to give a talk (or when you are applying to present at a conference), people want to see actual results, not research design. The American National Electoin Study (ANES) has made a nice step in this direction by setting up an online commons where users can offer comments about the design of proposed new question for the ANES. Lacking the vast resources of the ANES, I figured I’d turn instead to the one resource I did have at my disposal—this blog and its excellent readers—to see if anyone had any suggestions/critiques for an experiment I’m trying to tweak.

The experiment is designed to assess the relative importance of corruption in influencing vote choice as opposed to concerns about the state of the economy. Following the economic voting literature, we* conceive of “corruption voting” as having both potential pocketbook components (e.g., what is the effect of having actually had to pay bribes on your voting behavior?) and soicotropic components (e.g., what is the effect of one’s perceptions about the pervasiveness of corroption on one’s voting behavior?). In a paper that we will be presenting at the American Political Science Association’s Annual Meeting in Seattle a few weeks (Saturday, Sept 1 at 4:15 in Convention Center Room 308), we will report results from a traditional survey in one post-communist country showing much stronger support for “pocketbook corruption voting” than “sociotropic corruption voting”, both in terms of turnout and in terms of voting against the incumbent (i.e., having had to pay a bribe makes you less likely to turnout and less likely to vote for the incumbent). Given the predominence of sociotropic explanations in the economic voting literature and the received wisdom in post-communist countries that part of the reason incumbent governments get voted out of office so often is because people are fed up with overall levels of corruption, this was a pretty surprising finding.

We are now in the process of attempting to see if we can replicate this finding using an experimental research design. We’ve run the experiment once in a different post-communist country and have found results that again suggest pocketbook corruption concerns have more of an effect on voting than sociotropic corruption concerns. However, we have the opportunity to run this experiment in a second country, and we’re hoping to improve the experimental design. As I fear the vast majority of readers of even The Monkey Cage may not be interested in the nuts and bolts of experimental research design, the actual design of the experiment and the question I have about it are found after the break:

Here’s the experiment as designed, with the name of the country XXX’d out:

As you can see the “treatment” for the state of the economy is substantial: a shift from a worsened economy to an improved economy. The “treatment” for corruption, however, is much weaker: from a personal corruption experience (to the extent possible in a survey experiment…) to a perception of corruption. While the survey design allowed us to analyze the relative importance of this pocketbook vs. sociotropic corruption in influencing the voting decision, it is much less useful for analyzing the relative importance of corruption vs. economic conditions from an experimental standpoint because the strength of the treatment is so different.

So here’s one way to think of it: the economic treatment essentially goes from bad to good; the corruption treatment from one form of bad to another form of bad. So for the next version of the experiment, we want to add a “good” treatment to the corruption part of the equation as well. We will have 600 respondents, so we can make this a 3 X 2 experiment and still have 100 respondents in each cell. Here’s how we are thinking of designing the revised version of the experiment – everything will remain the same, but there will be three treatments instead of two. So it will read:

INSERT1 = “Last month, [NAME] had to spend half of his monthly salary to speed up the approval of permits for his business”
INSERT2 = “Last month, [NAME] heard that several city officials have taken bribes in exchange for government contracts”
INSERT3 = “[NAME] lives in a town with a mayor who has a strong reputation for fighting against corruption”

Any thoughts on “INSERT3”? (We’re going to keep the first two the same for comparative purposes.) Does it seem like a satisfactory “good” comparison in terms of corruption? Any concerns with it?

The kicker is that it would be most helpful if I heard back from you in the next 24 hours or so, but any feedback is of course useful. Feel free to respond in comments below or email me directly at joshua dot tucker at nyu dot edu.



* My co-authors on the corruption paper are Marko Klasnja and Kevin Deegan-Krause. My co-PI on the survey project is Ted Brader.

10 Responses to Request for Feedback on Experiment: Corruption and Voting

  1. idiot August 13, 2011 at 2:28 pm #

    I think it’s fine (and I also asked somebody else who cared about corruption about this question, and he said it was fine. To me, if a mayor has fought against corruption, then it suggests that the citizens living there wouldn’t have to deal with it.

  2. Scott August 13, 2011 at 2:54 pm #

    Is it possible that the treatment of reading about an individual having to personally pay bribes vs. an individual hearing about bribes also implies more prevalent and severe corruption? If so, you have a confound.

  3. Joshua Tucker August 13, 2011 at 2:59 pm #

    @Scott: Which one do you think of as more prevalent? I could see it going both ways – if it actually reached me, must be really prevalent. Or, well, it might have been only me, but now I hear it is more widespread than that.

    • Scott August 13, 2011 at 3:13 pm #

      I would guess that pocketbook is perceived as more prevalent. For example, if I were a victim of crime, I would imagine it is more prevalent than if I simply heard about it in the news. Might be worth testing this assumption on a convenience sample at least.

      • Joshua Tucker August 13, 2011 at 3:19 pm #

        I think we can at least test this in the survey data we have. Thanks for the suggestion.

  4. Solomon Kleinsmith August 13, 2011 at 3:46 pm #

    Great idea to ask for people to suggest question mods.

    I agree with the general idea above that, if there is widespread corruption allegations, more than a single example should be given and a statement regarding there being wider allegations should be made. If there has only been one allegation, that should also be made clear, as there is a huge difference between AN allegation, and widespread allegations over a period of time that paint a picture, so to speak.

    I’d also add that II. should have a similar range of options, as I. does.

    I really do despise questions like this, where you only have one or the other to choose, when reality almost never reflects a clear dichotomy like that. Perhaps something to the tune of:

    1. Almost entirely Economic Conditions
    2. Some of both, but more based on Economic Conditions
    3. About an even mix of both Economic Conditions and Concerns Related to Corruption
    4. Some of both, but more based on Concerns Related to Corruption
    5. Almost entirely Concerns Related to Corruption

    This sort of thing would give you much better data.

  5. Solomon Kleinsmith August 13, 2011 at 3:50 pm #

    This is a total tangent, but I would seriously consider installing a wordpress plugin by the name of “Subscribe To Comments”, so people who comment can check a box and an alert will

    When I added that to my blog several months ago, my comment reply rate shot up… almost geometrically. Really encourages more debate. If you need help with installing plugins, I’d be happy to walk you through it, it’s quite easy once you get the hang of it… love your blog, just wanna help!

    • Joshua Tucker August 13, 2011 at 3:54 pm #

      Thanks! John and Henry manage the blog, so I’ll let them know.

  6. Mark M. Fredrickson August 13, 2011 at 4:05 pm #

    As a complement to pocketbook vs. sociotropic economic evaluations, you may wish to compare types of corruption. Redlawsk and McCann (2005) argue that American voters see a difference in types of corruption, with corrupt acts being classified as either law-breaking or favoritism. In your context, the corruption conditions could be (1) bribe taking or (2) favoritism to cronies (which could be thought of as a negative bribe).

    Redlawsk, D. P. and McCann, J. A. (2005). Popular interpretations of ‘corruption’ and their partisan consequences. Political Behavior, 27:261–283.

  7. Jake Shapiro August 14, 2011 at 9:40 pm #

    You guys should talk to Luke Condra (Pitt) and Rikhil Bhavnani (UW Madison) who have designed a couple of field experiments to test almost this exact issue. Not sure where they are on fielding but they had plans to do in several very interesting developing countries.