Is theory getting lost in the “identification revolution”?

by Joshua Tucker on June 14, 2013 · 9 comments

in Methodology,Political Economy

The following is a guest post from Columbia University political scientist John Huber, and is a slightly modified version of a commentary that previously appeared in the newsletter of the Political Economy Section of the American Political Science Association.

*****

There is a powerful movement in social science emphasizing the importance of causal identification, of making valid causal inferences in empirical research.  A wide range of methods and approaches are being employed to help us figure out “what causes what” in politics, economics and their intersection, and although much of this research is in its relative infancy, the rapid progress social scientists are making to improve our understanding of how to approach the problem of causal identification should be embraced and celebrated.   At the same time, a laser focus on causal identification can create biases in the way we think about what constitutes a good question, in the claims we make about our work, and ultimately in how deeply we really understand social science phenomena.  It’s therefore useful to reflect a bit on the nature of these biases, and on how they might be shaping the way we go about our research.

The argument behind the “identification revolution” is well-rehearsed:  standard analyses of observational data, such as traditional multivariate regression with covariate adjustment, do not reveal the causal impact of variables because it is typically impossible with such approaches to understand the direction of causation, or to know if “effects” we attribute to some variable of interest are in fact due to some other unobserved variable that we have not measured.  We must therefore employ other approaches that allow random assignment of the causal variables of interest (such as field, laboratory or survey experiments), or at least that employ approaches to observational data that make causal inference possible (such as regression discontinuity models, instrumental variables, difference-in-difference models, or natural experiments).

Although the arguments underpinning the identification revolution are clearly correct from a methodological perspective, it is less obvious what the implications should be for how we proceed in efforts to understand social, economic and political  phenomena, and I worry there may be two unhelpful biases in how the on the identification revolution is influencing research strategies and agendas.  The first bias concerns the menu of questions we study.   Some “identificationists” take the strong position that social science research that cannot solve the identification problem is not worth doing, or at least is not worth publishing in leading journals.  If we move towards this position, we excessively narrow the range of questions we ask, and thus unnecessarily limit our understanding of the social processes we study.  One problem is that many things we care about – democracy, growth, institutions, diversity, inequality, wealth, violence, stability, rights, participation – cannot realistically be randomly assigned, and the extent to which the natural world presents us with causal identification opportunities can be quite limited.   Another problem is that many of these substantively important variables are embedded in dynamics of reciprocal causation with each other that will often frustrate the ambitions of even the most determined and talented “identificationists.”  Thus, good causal identification is not always possible on questions of central importance.

Does this mean we should not study such questions?  Sometimes research agendas reach a point where we won’t make much more useful progress until someone solves the identification problem.  The theories are well-developed, there exist no data limitations on how we describe empirical associations, and the traditional empirical methods have pushed observational data to their limits.  In these situations, further studies that leave unaddressed questions of causality seem a waste of time.  But the number of questions on which we’ve reached this point might be smaller than many imagine, and there is often much to be gained from working on questions for which we cannot see clear solutions to the identification problem.  Indeed, for many important questions, there is little clear theory, and providing one will be helpful in orienting empirical research.  Similarly, demonstrating the presence of previously unknown empirical associations can dramatically shape how we think about social phenomena, even if we can’t nail down causation.  It’s pretty impressive, for example, how often simple bivariate scatter plots make a lasting impact on how we think about the world around us.   Add the two together – theory and empirical association – and something very useful results, including making it possible to offer much better advice about what specific type of “identification study” is likely to yield the most useful insights.

But it’s hardly the case that theory is particularly important on questions where we haven’t figured out how to solve identification problems.  On the contrary, I think that one of the most important biases in how the identification revolution is unfolding concerns its distant relationship with theory.  It seems that developing clear theories, once a central pre-occupation in political economy, is now taking a back seat to developing convincing identification strategies.  Perhaps there’s a sense that we have plenty of theories and that the main challenge we face is to figure out which variables actually have a causal effect.  But this is wrong-headed — the very nature of research on causal identification requires a heightened rather diminished role for careful theorizing.

Many of the best strategies in causal identification essentially amount to case studies– a field experiment will be undertaken in a neighborhood during an election or a narrow set of villages; a natural experiment can be exploited as an opportunity for causal identification; a regression discontinuity strategy will be possible in a particular context.  It is the focus on the specific case that typically allows for convincing causal identification, and when done well, we can know with near certainty whether in the specific case, variable x has had a causal effect on variable y.   The problem, however, lies in situating the case, which is crucial if we are to draw useful inferences about causation.

Consider the example of how district magnitude – the number of representatives elected from a district – affects the number of political parties.   Suppose we want to understand whether a higher district magnitude leads to a larger number of parties, and we discover that something happened in Sweden that resulted in the random assignment of district magnitude for local elections.   We are able to exploit this to do a gold-plated causal identification study, and it reveals no effect of district magnitude on the party system. What can we learn from this?  That district magnitude has no effect on party systems?  The answer depends on how we situate the case of Sweden, which depends on theory.  In this example, leading theories argue that district magnitude should not affect party systems in especially homogenous societies (because the demand for parties in such societies is low), which helps us to situate the case (Sweden is very homogenous) and to think about how to interpret the null result.  And if we had thought about the theory carefully ex ante, we might have looked elsewhere for a case – if we had raised enough money to randomly assign electoral laws in the subnational governments of one country, for example, theory probably wouldn’t have directed us to do so in Sweden.  Of course, had the study occurred in a heterogeneous country and we’d found a positive effect, we’d still need a theory to guide interpretation – the positive effect may not imply, for example, that district magnitude generally leads to more parties.

Traditional regression-type methods with observational data play a crucial role in helping us to understand how to situate cases, and to learn inductively about how to build appropriate theories.  There exist famous papers of party systems that use such methods, for example, to make clear the presence of an interaction between district magnitude and ethnic heterogeneity.  Without such papers, we might be at sea when it comes to building theories that can explain the null results of the Swedish study.  In fact, though this is a matter of taste, I think we learn more about party systems in this example from one gold-plated traditional regression-type paper than we do from one gold-plated study in Sweden, even if the traditional approach leaves open the crucial question of causality.

Fortunately, the choice is not between the two empirical approaches. Traditional empirical approaches may play a useful role in helping us think about what a good theory might look like, but it is theory – not inductive empirical research – that enables us to convincingly situate and thus to understand the results of a gold-plated “causal identification” study, such as the hypothetical one in Sweden.  And I think we should worry that the importance of theory is getting lost in the “identification revolution.”  Too seldom do we see serious effort to use theory to motivate and situate cases.  And too often do we see conclusions from “causal identification case studies” with sweeping generalizations that over-claim – or even misjudge – what we can learn from the case.

I think it is inevitable that we will move toward a better dialogue between causal identification research and theory.  Although useful theorizing can take many different forms, my own bias is that formal theories will be especially important in this regard because they not only describe explicitly a logic explaining the circumstances under which some variable x should have a causal effect on y, but also because they can do so in a way that can generate multiple observable implications from a unified framework.  A formal theory, for example, can yield equilibrium predictions not only about the relationship between district magnitude and the number of parties, but also about how district magnitude affects the ideological locations of parties, or about how it affects the propensity for strategic voting.  This enables us not only to situate and interpret cases, but also to think about what types of causal identification research we should do in the first place.

If it’s true that the dialogue between formal theoretical research and causal identification research is not what it should be, the blame hardly lies solely in the laps of those doing the cutting-edge empirical research.  Too often, formal theorizing develops with indifference to (or in ignorance of) empirical research.  Theorists need to engage more directly the identification revolution, developing models explicitly directed at the topics studied in this research.  In so doing, theory must do more than provide plausible interpretations of the findings from specific cases – it must also help situate the cases, thereby providing arguments for the most fruitful avenues for subsequent empirical research.  What we need, then, are better synergies between theory building and causal identification studies.  The utility of each without the other is quite limited compared with what they can achieve together.

{ 9 comments }

Wonks Anonymous June 14, 2013 at 10:35 am

If you haven’t already read it, you might be interested in Prescott’s “Theory Ahead of Business Cycle Measurement”. The basic idea is that theoretical work suggests what data you’d want to look at.

Cyrus June 14, 2013 at 1:39 pm

Movement in this direction is well underway. Pick up the APSR today (or similar such journals in our sister disciplines) and you will see ample evidence of the fruitful marriage of good identification and theoretically grounded analyses. It took some time to get here, I think, because the implications of the identification revolution for ossified standard empirical practice in political science were so devastating. It made sense for attention to focus on such these novel realizations so that they could sink in. Hopefully no one is really arguing that we go back to heroic regression studies, but rather we understand what the identification revolution has taught about what reliable empirical practice requires. Now we can use those principles to get back to the job of theory building and testing, but on a more solid footing.

AZ June 14, 2013 at 6:32 pm

Bravo! What an excellent explication of an important point.

Gaurav June 14, 2013 at 7:24 pm

There are at least two perverse effects of what you call the “identification revolution” -
1. A variety of sub-par questions make it to top journals just because they have identification nailed down. (I do not take the stand that articles making causal claims shouldn’t have identification nailed down.)
2. Clear identification demands clear definition of treatment but it does not demand a particularly fine definition of it. So one can study effect of campaign ads or of introduction of Fox etc. but the treatment is too coarse for learning too much about theory.

Cyrus June 15, 2013 at 3:21 am

Gaurav – I don’t buy it. “Sub-par questions” sounds pretty subjective to me. If a paper at a top journal is “sub-par” by your standards, I am pretty sure you’ll find a sizable fraction in the discipline that would argue the opposite. For the second “perverse effect,” my own reading is that in well-identified studies, sometimes the treatment is too coarse, sometimes too tight, sometimes just right. When it’s too coarse, what we end up having to do is to apply techniques that use similar assumptions as those that one would have to apply from the get-go in a heroic regression study. When it’s too tight (Fox News), on the one hand we actually benefit from the clarity and tangibility of the finding; on the other hand, we do need to think hard about the particularities of our treatment, which is a productive theoretical exercise.

Tracy Lightcap June 15, 2013 at 12:14 pm

Cosma Shalizi’s new book has a lot of good things to say about this whole problem. See:

http://www.stat.cmu.edu/~cshalizi/uADA/13/lectures/ch22.pdf

Cosma favors good old fashioned directed graphs as the tool of choice to parse out causal relationships from observations. I’m no statistician, but I find his way forward more plausible as a strategy. My preference is always – always – to allow the patterns in the data to lead theoretical speculation. It’s the data that counts; the theory is only allows us to make sense of it and generate plausible (and often wrong) predictions about what we’ll find next. That’s of surpassing importance, of course, but only if the data leads. Real science is interactive between data patterns and theory; there usually isn’t any meaningful precedence in actual research.

absurdbeats June 15, 2013 at 12:48 pm

@Tracy Lightcap: Your comment puts me in mind of Francis Crick’s reminiscence on the false starts leading up to the discovery of the double helix of DNA:

‘You must remember, we ere trying to solve it with the minimum number of assumptions,’ Crick said. ‘There’s a perfectly sound reason—it isn’t just a matter of aesthetics or because we thought it was a nice game—why you should use the minimum of experimental data. The fact is, you remember, is that we knew that Bragg and Kendrew and Perutz had been misled my the experimental data. . . . The point is that evidence can be unreliable, and therefore you should use at little of it as you can. . . . I mean, people don’t realize that not only can such data be wrong in science, it can be misleading. There isn’t such a thing as a hard fact when you are trying to discover something. It’s only afterwards that the facts become hard.’

(From Horace Freeland Judson’s “The Eighth Day of Creation,” pp. 113-114)

Tracy Lightcap June 15, 2013 at 5:12 pm

Strange, since in the Double Helix, it was a lack of access to Rosalind Franklin’s observational data concerning the second form of DNA that held Watson and Crick up so much and the final building block for the end model was that it fit Chertoff (I think I have the name right) rules for the proportions of the five components of the molecule. I think what Crick is actually saying here is that the assumptions you use when modeling should be the minimum necessary to accommodate the evidence, largely because this makes it easier to model in the first place. Sort of like the “Rule of Three” in regression models, iow.

But I haven’t read Crick and he was a better theorist then Watson. It is interesting to note that Shalizi was also brought up a physicist, just like Crick. Obviously the experience cuts different ways with different people.

Jangle June 15, 2013 at 3:56 pm

Woooohoooooo! Go John you’re the best political scientist!!!

Comments on this entry are closed.

Previous post:

Next post: