Archive | Methodology

We are what we are studying

Anthropologist Marshall Sahlins writes:

When native Australians or New Guineans say that their totemic animals and plants are their kinsmen – that these species are persons like themselves, and that in offering them to others they are giving away part of their own substance – we have to take them seriously, which is to say empirically, if we want to understand the large consequences of these facts for how they organise their lives. The graveyard of ethnographic studies is strewn with the remains of reports which, thanks to anthropologists’ own presuppositions as to what constitutes empirical fact, were content to ignore or debunk the Amazonian peoples who said that the animals they hunted were their brothers-in-law, the Africans who described the way they systematically killed their kings when they became weak, or the Fijian chiefs who claimed they were gods.

My first thought was . . . wait a minute! Whazzat with “presuppositions as to what constitutes empirical fact”? That animal is or is not your brother-in-law, right? They’re either doing inter-species marriage in Amazonia or not, no?

But Sahlins does have something reasonable to say, and it’s relevant to my own research in political science. I’ll get to that in a moment, but first here’s Sahlins, continuing:

We have to follow the reasoning of those Australian Aboriginals for whom eating their own totem animals or plants would be something like incest or self-cannibalising, even as they ritually nourish and protect these species for other people’s use. We thus discover a society the opposite in principle of the bellicose state of nature that Hobbes posited as the primordial condition – an idea which is still too much with us. Of course the native Australians have known injurious disputes, most of them interpersonal. Yet instead of a Hobbesian ‘war of every man against every man’, each opposing others in his own self-interest, here is a society fundamentally organised on the premise of everyone giving himself to everyone.

In the earlier Germanic version of the natural science controversy, this human science alternative was called ‘understanding’, the implication being that the subject matter at issue was meaningfully or symbolically constructed, so that what was methodologically required was the penetration of its particular logic. The human scientist is not in a relation of a thinking person to a mute object of interest; rather, anthropologists and their like are of the same intellectual nature as the peoples they study: they are our alters and interlocutors. . . .

He then goes on to make some statements, with which I disagree, on the topic of natural science. But let’s forget about that and just go with the quote above. What struck me is the relevance of this “anthropological” mode of thinking to political science, where we must have understanding and sympathy for a wide spectrum of political opinions ranging from opposition to interracial marriage (supported by 46% of respondents in a recent poll of Mississippi Republican voters) to support for the nationalization of the means of production (still a popular position in many European countries, or so I’ve heard). As a political scientist studying public opinion, I have certain tools and academic experiences. But I am fundamentally the same kind of object as the people I am studying. It’s an obvious point but still worth remembering. This is the sort of thing that Dan Kahan writes about.

Continue Reading

Journal of Experimental Political Science

The American Political Science Association is coming out with a new journal:

The Journal of Experimental Political Science features research – be it theoretical, empirical, methodological, or some combination thereof – that utilizes experimental methods or experimental reasoning based on naturally occurring data. We define experimental methods broadly: research featuring random (or quasi-random) assignment of subjects to different treatments in an effort to isolate causal relationships between variables of interest. JEPS embraces all of the different types of experiments carried out as part of political science research, including survey experiments, laboratory experiments, field experiments, lab experiments in the field, natural and neurological experiments.

We invite authors to submit concise articles (around 2500 words) that immediately address the subject of the research (although in certain cases initial submissions can be longer than this limit with the understanding that if accepted the paper will be shortened within the word constraints). We do not require lengthy explanations regarding and justifications of the experimental method. Nor do we expect extensive literature reviews of pros and cons of the methodological approaches involved in the experiment unless the goal of the article is to explore these methodological issues. We expect readers to be familiar with experimental methods and therefore to not need pages of literature reviews to be convinced that experimental methods are a legitimate methodological approach. We also consider more lengthy articles in appropriate cases, as in the following examples: when a new experimental method or approach is being introduced and discussed, when a meta-analysis of existing experimental research is provided, or when new theoretical results are being evaluated through experimentation and the theoretical results are previously unpublished. Finally, we strongly encourage authors to submit null or inconsistent results from well-designed, executed, and analyzed experiments as well as replication studies of earlier experiments.

This looks good to me. There’s only one thing I’m worried about. Regular readers of the sister blog will be aware that there’s been a big problem in psychology, with the top journals publishing weak papers generalizing to the population based on Mechanical Turk samples and college students, lots of researcher degrees of freedom ensuring there will be no problem finding statistical significance, and with the sort of small sample sizes that ensure that any statistically significant finding will be noise, thus no particular reason to expect that patterns in the data will generalize to the larger population. A notorious recent example was a purported correlation between ovulation and political attitudes.

For some reason I seem to hear more about these sorts of papers in psycyhology than in poli sci (there was this paper by some political scientists, but it was not published in an actual poli sci journal).

Just to be clear: I’m not saying that the scientific claims being made in these papers are necessarily wrong, it’s just that these claims are not supported by the data. The papers are essentially exercises in speculation, “p=0.05” notwithstanding.

And I’m not saying that the authors of these papers are bad guys. I expect that they mostly just don’t know any better. They’ve been trained that “statistically significant” = real, and they go with that.

Anyway, I’m hoping this new journal of experimental political science will take a hard line and simply refuse to publish small-n experimental studies of small effects. Sometimes, of course, small-n is all you have, for example in a historical study of wars or economic depressions or whatever. But there you have to be careful to grapple with the limitations of your analyses. I’m not objecting to small-n studies of important topics. What I’m objecting to is fishing expeditions disguised as rigorous studies. In starting this new journal, we as a field just have to avoid the trap that the journal Psychological Science fell into, of seeming to feel an obligation to publish all sorts of iffy stuff that happened to combine headline-worthiness with (spurious) statistical significance.

P.S. I wrote this post last night and scheduled it to appear this morning. In the meantime, Josh posted more on this new journal. I hope it goes well.

Continue Reading

Yes, worry about generalizing from data to population. But multilevel modeling is the solution, not the problem

Screen Shot 2013-07-10 at 5.53.27 PM

A sociologist writes in:

Samuel Lucas has just published a paper in Quality and Quantity arguing that anything less than a full probability sample of higher levels in HLMs yields biased and unusable results. If I follow him correctly, he is arguing that not only are the SEs too small, but the parameter estimates themselves are biased and we cannot say in advance whether the bias is positive or negative.

Lucas has thrown down a big gauntlet, advising us throw away our data unless the sample of macro units is right and ignore the published results that fail this standard. Extreme.
Is there another conclusion to be drawn?
Other advice to be given?
A Bayesian path out of the valley?

The short answer is that I think Lucas is being unnecessarily alarmist. Continue Reading →

Continue Reading

Causal inference, extrapolating from sample to population

In a new paper titled “Does Regression Produce Representative Estimates of Causal Effects?”, Peter Aronow and Cyrus Samii write:

It is well-known that, with an unrepresentative sample, the estimate of a causal effect may fail to characterize how effects operate in the population of interest. What is less well understood is that conventional estimation practices for observational studies may produce the same problem even with a representative sample. Specifically, causal effects estimated via multiple regression differentially weight each unit’s contribution. The “effective sample” that regression uses to generate the causal effect estimate may bear little resemblance to the population of interest. The effects that multiple regression estimate may be nonrepresentative in a similar manner as are effects produced via quasi-experimental methods such as instrumental variables, matching, or regression discontinuity designs, implying there is no general external validity basis for preferring multiple regression on representative samples over quasi-experimental methods. We show how to estimate the implied multiple-regression weights for each unit, thus allowing researchers to visualize the characteristics of the effective sample. We then discuss alternative approaches that, under certain conditions, recover representative average causal effects. The requisite conditions cannot always be met.

They work within a poststratification-like framework, which I like, and I agree with their message. Here’s what I wrote on the topic a couple years ago:

It would be tempting to split the difference in the present debate [between proponents of field experiments and observational studies] and say something like the following: Randomized experiments give you accurate estimates of things you don’t care about; Observational studies give biased estimates of things that actually matter. The difficulty with this formulation is that inferences from observational studies also have to be extrapolated to correspond to the ultimate policy goals. Observational studies can be applied in many more settings than experiments but they address the same sort of specific micro-questions. . . . I recommend we learn some lessons from the experience of educational researchers, who have been running large experiments for decades and realize that, first, experiments give you a degree of confidence that you can rarely get from an observational analysis; and, second, that the mapping from any research finding—experimental or observational—is in effect an ongoing conversation among models, data, and analysis.

But that’s just words; Aronow and Samii back up their words with math, which is a good thing. I only have two minor comments on their paper:

1. Table 1 should be a graph. Use coefplot() or something like that. Do we really care that some variable has a mean of “47.58”?

2. I think the title is misleading in that it sets “regression” in opposition to designed experiments or natural experiments. Regression is typically the right tool to use when analyzing experimental or observational data. In either case, we are faced with the usual statistical problem of generalizing from sample to population.

Continue Reading

Conservative political analytics firm is hiring

Screen Shot 2013-06-21 at 10.17.12 AM

After posting that announcement about Civis Analytics, I wrote, “If a reconstituted Romney Analytics team is hiring, let me know and I’ll post that ad too.” Adam Schaeffer obliged:

Not sure about Romney’s team, but Evolving Strategies is looking for sharp folks who lean right:

Evolving Strategies is a political communications research firm specializing in randomized controlled experiments in the “lab” and in the “field.” ES is bringing a scientific revolution to free-market/conservative politics.

We are looking for people who are obsessive about getting things right and creative in their work. A ideal candidate will have a deep understanding of the academic literature in their field, highly developed skills, a commitment to academic rigor, but an intuitive understanding of practical political concerns and objectives as well.

We’re looking for new talent to help with our fast-growing portfolio in these areas:
High-level data processing, statistical analysis and modeling
Experimental design and execution

Helpful skills and experience include:
Experience designing and implementing social science experiments, online “lab” and “field.”
Extensive experience in statistical analysis and mastery of statistics software (R, Stata, etc.).
Extensive experience analyzing experimental and/or large datasets.
Extensive knowledge of academic research on political behavior/psychology.
Experience with or serious interest and knowledge of practical political activities; campaigns, issue-advocacy, etc.
Advanced degree (M.A., M.S., or Ph.D) in quantitative social sciences is preferred but not required with sufficient demonstration of skill and experience.

If interested, please send a C.V./resume and short cover-letter regarding:
Relevant skills and background
Interest in part-time consulting or full-time employment
Maximum hours per week available (indicate general availability and willingness to work longer hours on a short-term basis)
Hourly pay expectation or expected salary

Please send all questions and relevant materials to:

Evolving Strategies is an equal opportunity employer located in Northern Virginia. Working remotely is possible for all part time consulting and some salaried positions. Salary is commensurate with experience.

Continue Reading

Is theory getting lost in the “identification revolution”?

The following is a guest post from Columbia University political scientist John Huber, and is a slightly modified version of a commentary that previously appeared in the newsletter of the Political Economy Section of the American Political Science Association.


There is a powerful movement in social science emphasizing the importance of causal identification, of making valid causal inferences in empirical research.  A wide range of methods and approaches are being employed to help us figure out “what causes what” in politics, economics and their intersection, and although much of this research is in its relative infancy, the rapid progress social scientists are making to improve our understanding of how to approach the problem of causal identification should be embraced and celebrated.   At the same time, a laser focus on causal identification can create biases in the way we think about what constitutes a good question, in the claims we make about our work, and ultimately in how deeply we really understand social science phenomena.  It’s therefore useful to reflect a bit on the nature of these biases, and on how they might be shaping the way we go about our research.

The argument behind the “identification revolution” is well-rehearsed:  standard analyses of observational data, such as traditional multivariate regression with covariate adjustment, do not reveal the causal impact of variables because it is typically impossible with such approaches to understand the direction of causation, or to know if “effects” we attribute to some variable of interest are in fact due to some other unobserved variable that we have not measured.  We must therefore employ other approaches that allow random assignment of the causal variables of interest (such as field, laboratory or survey experiments), or at least that employ approaches to observational data that make causal inference possible (such as regression discontinuity models, instrumental variables, difference-in-difference models, or natural experiments).

Although the arguments underpinning the identification revolution are clearly correct from a methodological perspective, it is less obvious what the implications should be for how we proceed in efforts to understand social, economic and political  phenomena, and I worry there may be two unhelpful biases in how the on the identification revolution is influencing research strategies and agendas.  The first bias concerns the menu of questions we study.   Some “identificationists” take the strong position that social science research that cannot solve the identification problem is not worth doing, or at least is not worth publishing in leading journals.  If we move towards this position, we excessively narrow the range of questions we ask, and thus unnecessarily limit our understanding of the social processes we study.  One problem is that many things we care about – democracy, growth, institutions, diversity, inequality, wealth, violence, stability, rights, participation – cannot realistically be randomly assigned, and the extent to which the natural world presents us with causal identification opportunities can be quite limited.   Another problem is that many of these substantively important variables are embedded in dynamics of reciprocal causation with each other that will often frustrate the ambitions of even the most determined and talented “identificationists.”  Thus, good causal identification is not always possible on questions of central importance.

Does this mean we should not study such questions?  Sometimes research agendas reach a point where we won’t make much more useful progress until someone solves the identification problem.  The theories are well-developed, there exist no data limitations on how we describe empirical associations, and the traditional empirical methods have pushed observational data to their limits.  In these situations, further studies that leave unaddressed questions of causality seem a waste of time.  But the number of questions on which we’ve reached this point might be smaller than many imagine, and there is often much to be gained from working on questions for which we cannot see clear solutions to the identification problem.  Indeed, for many important questions, there is little clear theory, and providing one will be helpful in orienting empirical research.  Similarly, demonstrating the presence of previously unknown empirical associations can dramatically shape how we think about social phenomena, even if we can’t nail down causation.  It’s pretty impressive, for example, how often simple bivariate scatter plots make a lasting impact on how we think about the world around us.   Add the two together – theory and empirical association – and something very useful results, including making it possible to offer much better advice about what specific type of “identification study” is likely to yield the most useful insights.

Continue Reading →

Continue Reading

Question wording and changing attitudes in acceptance of surveillance

John just posted some survey results comparing attitudes about secret National Security Agency wiretapping, comparing polls in 2006 and 2013. At first glance, support for the surveillance seems slightly higher than before, with 51% supporting it in 2006, and 56% supporting it now.

But look carefully at the questions:

In 2006: “secretly listening . . . without court approval”

In 2013: “getting secret court orders . . .”

So, more people support wiretapping now—-but the survey stipulates that the NSA got court orders. Sure, they’re “secret” court orders, but it means that a judge is somewhere in the loop. In contrast, the 2006 poll asked about extrajudicial wiretapping.

On the other direction, the 2013 question refers to “millions of Americans,” whereas the 2006 question asks about a more restricted class: “people suspected of terrorist involvement.”

I don’t know how important the question wording is; maybe people are just giving their gut reactions to recent headlines. On a substantive level, though, there’s a difference between tapping millions of phones vs. monitoring terrorist suspects, and there’s a difference between court order and no court order. I don’t know how I would respond to the poll now, and I don’t know how I’d have responded in 2006.

In his post, John also notes that attitudes are partisanly skewed, with a combination of two factors: (a) Members of the president’s party are more supportive than members of the opposition party, and (b) averaging the surveys from both years, Republicans are generally more supportive of surveillance than Democrats are.

Given the murkiness of the issue, it seems perfectly rational for people to be more supportive of secret government power when they trust the people running the government. (This is not intended to contradict John’s post in any way, just to elaborate on it.)

Continue Reading