Is theory getting lost in the “identification revolution”?

by Joshua Tucker on June 14, 2013 · 9 comments

in Methodology,Political Economy

The following is a guest post from Columbia University political scientist John Huber, and is a slightly modified version of a commentary that previously appeared in the newsletter of the Political Economy Section of the American Political Science Association.

*****

There is a powerful movement in social science emphasizing the importance of causal identification, of making valid causal inferences in empirical research.  A wide range of methods and approaches are being employed to help us figure out “what causes what” in politics, economics and their intersection, and although much of this research is in its relative infancy, the rapid progress social scientists are making to improve our understanding of how to approach the problem of causal identification should be embraced and celebrated.   At the same time, a laser focus on causal identification can create biases in the way we think about what constitutes a good question, in the claims we make about our work, and ultimately in how deeply we really understand social science phenomena.  It’s therefore useful to reflect a bit on the nature of these biases, and on how they might be shaping the way we go about our research.

The argument behind the “identification revolution” is well-rehearsed:  standard analyses of observational data, such as traditional multivariate regression with covariate adjustment, do not reveal the causal impact of variables because it is typically impossible with such approaches to understand the direction of causation, or to know if “effects” we attribute to some variable of interest are in fact due to some other unobserved variable that we have not measured.  We must therefore employ other approaches that allow random assignment of the causal variables of interest (such as field, laboratory or survey experiments), or at least that employ approaches to observational data that make causal inference possible (such as regression discontinuity models, instrumental variables, difference-in-difference models, or natural experiments).

Although the arguments underpinning the identification revolution are clearly correct from a methodological perspective, it is less obvious what the implications should be for how we proceed in efforts to understand social, economic and political  phenomena, and I worry there may be two unhelpful biases in how the on the identification revolution is influencing research strategies and agendas.  The first bias concerns the menu of questions we study.   Some “identificationists” take the strong position that social science research that cannot solve the identification problem is not worth doing, or at least is not worth publishing in leading journals.  If we move towards this position, we excessively narrow the range of questions we ask, and thus unnecessarily limit our understanding of the social processes we study.  One problem is that many things we care about – democracy, growth, institutions, diversity, inequality, wealth, violence, stability, rights, participation – cannot realistically be randomly assigned, and the extent to which the natural world presents us with causal identification opportunities can be quite limited.   Another problem is that many of these substantively important variables are embedded in dynamics of reciprocal causation with each other that will often frustrate the ambitions of even the most determined and talented “identificationists.”  Thus, good causal identification is not always possible on questions of central importance.

Does this mean we should not study such questions?  Sometimes research agendas reach a point where we won’t make much more useful progress until someone solves the identification problem.  The theories are well-developed, there exist no data limitations on how we describe empirical associations, and the traditional empirical methods have pushed observational data to their limits.  In these situations, further studies that leave unaddressed questions of causality seem a waste of time.  But the number of questions on which we’ve reached this point might be smaller than many imagine, and there is often much to be gained from working on questions for which we cannot see clear solutions to the identification problem.  Indeed, for many important questions, there is little clear theory, and providing one will be helpful in orienting empirical research.  Similarly, demonstrating the presence of previously unknown empirical associations can dramatically shape how we think about social phenomena, even if we can’t nail down causation.  It’s pretty impressive, for example, how often simple bivariate scatter plots make a lasting impact on how we think about the world around us.   Add the two together – theory and empirical association – and something very useful results, including making it possible to offer much better advice about what specific type of “identification study” is likely to yield the most useful insights.

But it’s hardly the case that theory is particularly important on questions where we haven’t figured out how to solve identification problems.  On the contrary, I think that one of the most important biases in how the identification revolution is unfolding concerns its distant relationship with theory.  It seems that developing clear theories, once a central pre-occupation in political economy, is now taking a back seat to developing convincing identification strategies.  Perhaps there’s a sense that we have plenty of theories and that the main challenge we face is to figure out which variables actually have a causal effect.  But this is wrong-headed—the very nature of research on causal identification requires a heightened rather diminished role for careful theorizing.

Many of the best strategies in causal identification essentially amount to case studies– a field experiment will be undertaken in a neighborhood during an election or a narrow set of villages; a natural experiment can be exploited as an opportunity for causal identification; a regression discontinuity strategy will be possible in a particular context.  It is the focus on the specific case that typically allows for convincing causal identification, and when done well, we can know with near certainty whether in the specific case, variable x has had a causal effect on variable y.   The problem, however, lies in situating the case, which is crucial if we are to draw useful inferences about causation.

Consider the example of how district magnitude – the number of representatives elected from a district – affects the number of political parties.   Suppose we want to understand whether a higher district magnitude leads to a larger number of parties, and we discover that something happened in Sweden that resulted in the random assignment of district magnitude for local elections.   We are able to exploit this to do a gold-plated causal identification study, and it reveals no effect of district magnitude on the party system. What can we learn from this?  That district magnitude has no effect on party systems?  The answer depends on how we situate the case of Sweden, which depends on theory.  In this example, leading theories argue that district magnitude should not affect party systems in especially homogenous societies (because the demand for parties in such societies is low), which helps us to situate the case (Sweden is very homogenous) and to think about how to interpret the null result.  And if we had thought about the theory carefully ex ante, we might have looked elsewhere for a case – if we had raised enough money to randomly assign electoral laws in the subnational governments of one country, for example, theory probably wouldn’t have directed us to do so in Sweden.  Of course, had the study occurred in a heterogeneous country and we’d found a positive effect, we’d still need a theory to guide interpretation – the positive effect may not imply, for example, that district magnitude generally leads to more parties.

Traditional regression-type methods with observational data play a crucial role in helping us to understand how to situate cases, and to learn inductively about how to build appropriate theories.  There exist famous papers of party systems that use such methods, for example, to make clear the presence of an interaction between district magnitude and ethnic heterogeneity.  Without such papers, we might be at sea when it comes to building theories that can explain the null results of the Swedish study.  In fact, though this is a matter of taste, I think we learn more about party systems in this example from one gold-plated traditional regression-type paper than we do from one gold-plated study in Sweden, even if the traditional approach leaves open the crucial question of causality.

Fortunately, the choice is not between the two empirical approaches. Traditional empirical approaches may play a useful role in helping us think about what a good theory might look like, but it is theory – not inductive empirical research – that enables us to convincingly situate and thus to understand the results of a gold-plated “causal identification” study, such as the hypothetical one in Sweden.  And I think we should worry that the importance of theory is getting lost in the “identification revolution.”  Too seldom do we see serious effort to use theory to motivate and situate cases.  And too often do we see conclusions from “causal identification case studies” with sweeping generalizations that over-claim – or even misjudge – what we can learn from the case.

I think it is inevitable that we will move toward a better dialogue between causal identification research and theory.  Although useful theorizing can take many different forms, my own bias is that formal theories will be especially important in this regard because they not only describe explicitly a logic explaining the circumstances under which some variable x should have a causal effect on y, but also because they can do so in a way that can generate multiple observable implications from a unified framework.  A formal theory, for example, can yield equilibrium predictions not only about the relationship between district magnitude and the number of parties, but also about how district magnitude affects the ideological locations of parties, or about how it affects the propensity for strategic voting.  This enables us not only to situate and interpret cases, but also to think about what types of causal identification research we should do in the first place.

If it’s true that the dialogue between formal theoretical research and causal identification research is not what it should be, the blame hardly lies solely in the laps of those doing the cutting-edge empirical research.  Too often, formal theorizing develops with indifference to (or in ignorance of) empirical research.  Theorists need to engage more directly the identification revolution, developing models explicitly directed at the topics studied in this research.  In so doing, theory must do more than provide plausible interpretations of the findings from specific cases – it must also help situate the cases, thereby providing arguments for the most fruitful avenues for subsequent empirical research.  What we need, then, are better synergies between theory building and causal identification studies.  The utility of each without the other is quite limited compared with what they can achieve together.

{ 9 comments }

Comments on this entry are closed.

Previous post:

Next post: