Kyle Peyton writes:
I’m passing you this recent news article by Ewen Callaway in the hope that you will make a comment about the methodology on your blog. It’s generated some back and forth between the economics and science communities.
I [Peyton] am very sceptical of the reductive approach taken by the economics profession generally, and the normative implications this kind of research generates. For example, p. 7 of the working paper states: “…[according to our model] decreasing the diversity of the most diverse country in the sample (Ethopia) by 1 percentage point would raise its income per capita by 21 percent”. Understandably, this piece is couched in assumptions that would take hours to pick apart, but their discussion of the approach belies the uncertainty involved. The main response by the authors in defense is that genetic diversity is a ‘proxy variable’. This is a common assertion, but I find it really infuriating. I happen to drink coffee most days, which correlates with my happiness. So coffee consumption is a ‘proxy’ for my happiness. Therefore, I can put it in a regression and predict the relationship between my happiness and the amount of times I go to the bathroom. Ergo universal conclusions: ‘relieving yourself improves mental well being.’ New policy – you should relieve yourself atleast 2 times per day in order to maintain high levels of emotional well being. I know this sound like a South Park episode, but I have heard far worse.
But let’s put the normative implication aside — what can we learn from star gazing at the tables in this paper?
Here’s the background. Two economics professors, Quamrul Ashraf and Oded Galor, wrote a paper, “The Out of Africa Hypothesis, Human Genetic Diversity, and Comparative Economic Development,” that is scheduled to appear in the American Economic Review. As Peyton has indicated, the paper is pretty silly and I’m surprised it was accepted in such a top journal. Economists can be credulous but I’d expect better from them when considering economic development, which is one of their central topics. Ashraf and Galor have, however, been somewhat lucky in their enemies, in that they’ve been attacked by a bunch of anthropologists who have criticized them on political as well as scientific grounds. This gives the pair of economists the scientific and even moral high ground, in that they can feel that, unlike their antagonists, they are the true scholars, the ones pursuing truth wherever it leads them, letting the chips fall where they may.
The real issue for me is that the chips aren’t quite falling the way Ashraf and Galor think they are. Let’s start with the claims on page 7 of their paper:
Once institutional, cultural, and geographical factors are accounted for, [the fitted regression] indicates that: (i) increasing the diversity of the most homogenous country in the sample (Bolivia) by 1 percentage point would raise its income per capita in the year 2000 CE by 41 percent, (ii) decreasing the diversity of the most diverse country in the sample (Ethiopia) by 1 percentage point would raise its income per capita by 21 percent.
I think “CE” is academic jargon for what we call “A.D.” in English (or Latin, whatever), and strictly speaking the above bit is not a claim at all, it’s just an interpretation of their regression coefficients. But it clearly is a claim, in that the authors want us to take these examples seriously.
So let’s take them seriously. What would it mean to increase Bolivia’s diversity by 1 percentage point? I assume that would mean adding some white people to the country. What kind of white person would go to Bolivia? Probably someone rich enough to increase the country’s income per capita. Hey—it works! What if some poor people from Ethiopia were taken to Bolivia? They’d increase the country’s ethnic diversity too, but I don’t see them increasing its per-capita income by 41 percent. But that’s ok, nobody’s suggesting filling Bolivia with poor Africans.
OK, what about Ethiopia? How do you make it less diverse? I guess you’d have to break it up into a bunch of little countries, each of which is ethnically pure. Is that possible? I don’t actually know. If you can’t do that, you’d need to throw in lots of people with less genetic diversity. Maybe, hmmm, I dunno, a bunch of whites or Asians? What sort of whites or Asians might go to Ethiopia? Not the poorest ones, certainly: why would they want to go to a poor country in the first place? Maybe some middle-income or rich ones (if the country could be safe enough, or if there’s a sense there’s money to be made). And, there you go, per-capita income goes up again.
So I don’t see it. It’s true that later on page 7 the authors try to wriggle out of this one:
Reassuringly, the highly significant and stable hump-shaped effect of genetic diversity on income per capita in the year 2000 CE is not an artifact of postcolonial migrations towards prosperous countries and the concomitant increase in ethnic diversity in these economies. The hump-shaped e§ect of genetic diversity remains highly signiÖcant and the optimal diversity estimate remains virtually intact if the regression sample is restricted to (i) non-OECD economies (i.e., economies that were less attractive to migrants), (ii) non-Neo-European countries (i.e., excluding the U.S., Canada, Australia, and New Zealand), (iii) non-Latin American countries, (iv) non-Sub-Saharan African countries, and, perhaps most importantly, (v) countries whose indigenous population is larger than 97 percent of the entire population (i.e., under conditions that virtually eliminate the role of migration in contributing to diversity).
I don’t buy it. I’m not saying their central point is wrong—it’s basically a twist on the classic “why are some countries so poor” question—but the extrapolations that they give themselves reveal the problems with their interpretation of the regression model. The way you make Bolivia more diverse is by adding more white people. It’s fine to study these things but you have to think about what your models mean.
Everybody wants to be Jared Diamond, that’s the problem.
OK, if all this is the case, what went wrong, and how could Ashraf and Galor have done better? I think the way to go is to start with the big pattern they noticed: the most genetically diverse countries (according to their measure) are in east Africa, and they’re poor. The least genetically diverse countries are remote undeveloped places like Bolivia and are pretty poor. Industrialized countries are not so remote (thus they have some diversity) but they’re not filled with east Africans (thus they’re not extremely genetically diverse). From there, you can look at various subsets of the data and perform various side analysis, as the authors indeed do for much of their paper.
The problem is closely related to their paper appearing in a top journal. The way I see this work, the authors have an interesting idea and want to explore it. But exploration won’t get you published in the American Economic Review. Instead of the explore-and-study paradigm, Ashraf and Galor are going with assert-and-defend. They make a very strong claim and keep banging on it, defending their claim with a bunch of analyses to demonstrate its robustness. I have no problem with robustness studies (recall that I was upset about some claims about age and happiness because I had difficulty replicating them with new data), but I don’t think this lets you off the hook of having to think carefully about causal claims. And presenting tables of numbers to three (meaningless) decimal places doesn’t help either.
High-profile social science research aims for proof, not for understanding—and that’s a problem. The incentives favor bold thinking and innovative analysis, and that part is great. But the incentives also favor silly causal claims. In many social sciences, it’s not enough to notice an interesting pattern and explore it (as we did in our Red State Blue State book). Instead, you’re supposed to make a strong causal claim even in a context where it makes little sense.