In a new paper titled “Does Regression Produce Representative Estimates of Causal Effects?”, Peter Aronow and Cyrus Samii write:
It is well-known that, with an unrepresentative sample, the estimate of a causal effect may fail to characterize how effects operate in the population of interest. What is less well understood is that conventional estimation practices for observational studies may produce the same problem even with a representative sample. Specifically, causal effects estimated via multiple regression differentially weight each unit’s contribution. The “effective sample” that regression uses to generate the causal effect estimate may bear little resemblance to the population of interest. The effects that multiple regression estimate may be nonrepresentative in a similar manner as are effects produced via quasi-experimental methods such as instrumental variables, matching, or regression discontinuity designs, implying there is no general external validity basis for preferring multiple regression on representative samples over quasi-experimental methods. We show how to estimate the implied multiple-regression weights for each unit, thus allowing researchers to visualize the characteristics of the effective sample. We then discuss alternative approaches that, under certain conditions, recover representative average causal effects. The requisite conditions cannot always be met.
They work within a poststratification-like framework, which I like, and I agree with their message. Here’s what I wrote on the topic a couple years ago:
It would be tempting to split the difference in the present debate [between proponents of field experiments and observational studies] and say something like the following: Randomized experiments give you accurate estimates of things you don’t care about; Observational studies give biased estimates of things that actually matter. The difficulty with this formulation is that inferences from observational studies also have to be extrapolated to correspond to the ultimate policy goals. Observational studies can be applied in many more settings than experiments but they address the same sort of specific micro-questions. . . . I recommend we learn some lessons from the experience of educational researchers, who have been running large experiments for decades and realize that, first, experiments give you a degree of confidence that you can rarely get from an observational analysis; and, second, that the mapping from any research finding—experimental or observational—is in effect an ongoing conversation among models, data, and analysis.
But that’s just words; Aronow and Samii back up their words with math, which is a good thing. I only have two minor comments on their paper:
1. Table 1 should be a graph. Use coefplot() or something like that. Do we really care that some variable has a mean of “47.58”?
2. I think the title is misleading in that it sets “regression” in opposition to designed experiments or natural experiments. Regression is typically the right tool to use when analyzing experimental or observational data. In either case, we are faced with the usual statistical problem of generalizing from sample to population.