More on those “Psychological Science” papers (menstrual cycles and political attitudes, biceps size and political attitudes)

by Andrew Gelman on May 30, 2013 · 22 comments

in Campaigns and elections,Public opinion,Science

I was thinking some more about those two recent studies (see here and here) published in Psychological Science, “the highest ranked empirical journal in psychology.”

Just to relive these for a moment, the papers are:

“The Ancestral Logic of Politics: Upper-Body Strength Regulates Men’s Assertion of Self-Interest Over Economic Redistribution”: 3 surveys, two of which were of college students, no actual direct measures of upper-body strength, regulation, assertion, or self-interest.

“The Fluctuating Female Vote: Politics, Religion, and the Ovulatory Cycle”: claims of implausibly huge within-person effects, no within-person measurements at all.

I’m pretty sure that neither of these papers would’ve had a chance of being accepted in the APSR or AJPS, the two leading journals in our field. I have no idea how far down you’d have to go to get such a paper published in a political science journal.

Just to be clear: I have no problem with such papers being published in a repository such as Plos-One (as was done by this later-debunked paper by some political scientists). Plos-One is a great place to put some shaky empirical work paired with speculative conclusions. You get it out there, make your data public, and other people can take a look. It’s in Plos-One, so savvy journalists know it’s speculation. Win-win.

If this stuff is being published in the top journal in your field, though, we’re in trouble.

I certainly don’t think political science is perfect, but the field seems to be pretty good at reserving space in the top journals for serious work. I’d say that maybe we need a journal for speculations, but (a) maybe such journals already exists, and (b) there’s always Plos-One, which seems to have the potential for playing a useful role as a sort of super-Arxiv across all disciplines.

The thing that bothers me about these various mini-papers that get so much attention is that they’re hyped beyond all recognition. Check out this official press release from the Association for Psychological Science (which, recall, split off several years ago from the American Psychological Association because they felt the ASA was not focused enough on scientific research):

Screen Shot 2013-05-30 at 7.15.00 AM

Of course we can’t criticize the APS from running a press release, given that the paper was accepted in their journal. But the press release itself is a mix of caveman speculation and a misleading description of the research. It’s misleading to speak of “wealthy men” when two of the surveys were of college students). It’s also misleading to make statements about how “individuals reason” based on a few survey questions about issue attitudes; that one is particularly embarrassing coming from psychologists (indeed, the keywords in the press release include “Cognitive Processes” and “Judgment,” even though there’s nothing in the data about cognitive processes or judgment). There’s also the crude and politically naive association of an attitude on redistribution with “self-interest.” Suppose young man comes from a family with low socioeconomic status with conservative economic views (actually, I don’t know how many low-SES students or people with conservative economic views they will get in their sample of University of California students, but that’s another story), then he might well view it to be not in his self-interest to support economic redistribution.

These authors are psychologists, there’s no reason for them to know much about political science, any more than I should be expected to know much about macroeconomics (despite its relevance to some of my research), but this still seems pretty bad to me. At least I’d hope they’d show some humility about the subtleties of politics, in the way that John Jost and Jon Haidt do from their quite different perspectives in their studies of political attitudes.

There’s also a political angle, in that these sorts of studies often seem to be used to support a wife-in-the-kitchen agenda. Blogger Echidne gives a hilariously-horrible example from Fox News:

Erick Erickson, one of Fox’s newest contributors, was troubled by female breadwinners and claimed that people who defend them are “anti-science.” Erickson told viewers:

When you look at biology, look at the natural world, the roles of a male and female in society, and the other animals, the male typically is the dominant role. The female, it’s not antithesis, or it’s not competing, it’s a complimentary role. We as people in a smart society have lost the ability to have complimentary relationships in nuclear families, and it’s tearing us apart.

Remember that the topic is women who earn money for their families. So Erickson seems to be arguing that no female animal goes out to get food ever, that it’s the male lions which feed the pride and so on, and that the female wolves never go out to hunt.

That is all total rubbish. In fact, I can’t think of any mammal where the female stays in the nest or lair with the young and the male goes out and brings all the food home. If that happens, at least among mammals, it is extremely rare. My suspicion is that single mothers are much more common among mammals than that alternative fable. Indeed, chimpanzees seem to have the single mother system.

What these four men are upset about is the fear that the traditional gender roles are breaking down. They like those gender roles because they like to be dominant.

But in most ways the traditional gender roles aren’t even that traditional, because very few people in the olden days could live like the Victorian images of a bourgeois nuclear family. Farm-wives worked, wives of artisans worked and so on.

Contrary to stereotype, in this case it’s the man who is babbling and the feminist who has the common sense.

Ok, this happens in both directions. Not so much in evolutionary psychology, I think, but there is a left-wing equivalent, of sorts, in various cross-national comparisons that get published in journals of public health. The basic idea is that good things tend to go together. There are some countries that are richer, more civilized, have nicer governments that spend more money on social welfare, have happier and more educated citizens, etc.; and other countries that are poor, dangerous, etc etc. As a result it’s not hard to show a correlation between, for example, social security spending and longevity, or whatever.

As with the evolutionary psychology studies, I’m not saying that these claims made in these cross-national correlation studies are wrong, merely that the evidence has problems and the claims can be overstated. And they get published in top journals because they are important. Actually, I’m not so bothered by these studies because the topics are important (more so, in my opinion, than claims about ovulation and voting). Similarly, I’m not so bothered by observational studies of the macroeconomy such as Reinhardt and Rogoff’s (implementation details aside); when a topic is important, you go with what you have, you can’t wait for identification.

{ 22 comments }

Jonathan Baron May 30, 2013 at 10:37 am

Psychological Science is notorious for publishing studies with “surprising” conclusions, i.e., bold claims that are likely to be false on the basis of prior personal probabilities of those who know the field, hence conclusions that are in fact likely to be false despite the reported evidence for them. In the past, and perhaps still in some areas, many of these articles were the result of p-hacking, selective reporting of studies that worked, and incorrect data analysis. I have noticed some recent improvement in my own field (judgment and decision making), and, incidentally, fewer papers in this area, perhaps a good thing. Only recently has the journal been at all willing to publish criticisms of previously published papers, and it is still apparently reluctant. Part of the reason it is so prestigious and widely read it is that it is sent to all members of the society that publishes it, which is a huge society. The other major psychology society (APA) does not send any empirical journals to its members unless they pay extra for subscriptions to these journals. And it does occasionally publish good, believable, interesting papers. Still, it sounds from your comments that it still has a ways to go in the direction of cleaning up our act.

I must differ about political science. It is not just a matter of higher standards but also arbitrary standards. For example, in my experience, studies of (or grant proposals concerning) general psychological issues about political judgment or attitudes, general in the sense of meant to apply to humans in general rather than Americans, are rejected because the subjects are not a random sample of Americans.

Aoi May 30, 2013 at 12:30 pm

My impression is that even within psychology, evolutionary psychology is somewhat controversial. Not because people don’t believe in evolution. But you can’t really do a direct test of any theory that presupposes that a multi-millenia-long process affects modern attitudes/behaviors. The closest you could get would be a truly worldwide, cross-cultural sample, and short of that (e.g., say, a survey or experimental manipulation of college students) anything you attribute to evolution could just as easily (and probably much more likely) be a product of more proximate social forces.

Jonathan Baron May 30, 2013 at 12:49 pm

Evolutionary psychology is indeed controversial. A lot of it is science fiction, just-so stories about how some behavior might have been adaptive at some time. Moreover, it is a subclass of the broader category of “functionalism”, which holds that behavior can be explained because it serves some purpose or function. This include a lot of learning theory, developmental psychology, cultural evolution, or even the results of people’s reflective thought about how to achieve some goal. These are thus all alternative explanations to that of biological evolution. And, of course, some behavioral phenomena might just serve no function at all. (Hyperbolic discounting comes to mind, putting aside the just-so stories about how it might.) The one area where I think that evolutionary psychology has enlightened us – there may be others – is the study of sexual behavior.

Aldous May 30, 2013 at 2:18 pm

“My impression is that even within psychology, evolutionary psychology is somewhat controversial. ”

“The one area where I think that evolutionary psychology has enlightened us – there may be others – is the study of sexual behavior.”

There is a connection here. There are entire DEPARTMENTS in the halls of the academy who believe that gender is entirely a social construction and that we are blank slates at birth.

Jacob H. May 31, 2013 at 9:45 am

My sense is that the Evolutionary Psychology worth reading begins and ends with Robert Trivers’ 1970s papers.

Jonathan Baron May 31, 2013 at 10:03 am

I am not an expert on evolutionary psychology. I study moral judgment. A few evolutionary psychologists have argued that certain patterns of moral judgment are the result of evolved cognitive modules. I find this implausible, in the light of lack of any good evidence favoring this hypothesis over the many (more plausible) alternatives. (Cosmides and Tooby have favored such explanations, at least in other domains like reasoning. The problems there are similar.) However, I do think that there is more interesting stuff in evolutionary psychology aside from Trivers (who, I admit, is very interesting, if perhaps also wrong), specifically the work on sexual preferences. The best work I know – which is surely just a random sample but it is pretty good – is that of Douglas Kenrick and (especially) J. Michael Bailey. Bailey (last time I looked) is not a “true believer”. He has a sophisticated view of where evolutionary thinking can and cannot help.

Echidne June 1, 2013 at 4:22 am

The one area where I think that evolutionary psychology has enlightened us – there may be others – is the study of sexual behavior.
My answer to that would be yes and no. This is because the evolutionary psychology research still uses today’s data and that data has alternative explanations which also need to be considered and tested.

For one example of how things may change by a rather small change in study design, note this earlier study using speed dating as a proxy for ancestral mate selection, then see what happens when the rule who asks was changed in speed-dating studies.

Likewise, international surveys about what men and women value in a potential mate show differences between countries, and there is even some question whether the interest gap between men and women, when it comes to casual sex, is as large as the now-classic (and flawed) Buss study found.

My apologies for linking to popularized summaries of the studies.

But I agree that evolutionary psychology has allowed us to ask interesting questions. What the final correct answers are still remains to be seen. I think the research has unnecessarily ignored more proximate explanations for some phenomena. For instance, casual sex with strangers has rather different risk factors (especially in pregnancy and the risk of sexual violence) for heterosexual men and women, so even if their desires didn’t differ, the average behavior by gender probably would.

This doesn’t mean that there might not be a gender difference in the innate desire for casual sex, simply that the alternatives should also be considered.

Jonathan Baron June 1, 2013 at 7:32 pm

You are not referring to the findings that impressed me as interesting, and derived from evolutionary theory. Kenrick, for example, finds that male sexual preferences are more age-specific than are female preferences, cross-culturally. Bailey has a whole set of really interesting results about homosexuality.

I am no fan of evolutionary psychology. I think the attempt to replace psychology with biology is terribly misguided, and evolutionary psychology is a symptom. (All this brain talk is the bigger problem.) I was trying to be open-minded.

But the “nature/nurture” question is somewhat different, and there I think that, whether human characteristics were selected because they were adaptive or not, many of these characteristics are affected by “nature”. I will stop replying to this discussion now. I supplied my email address for those who want to continue outside of this forum.

afewthings May 30, 2013 at 1:23 pm

first–one of the authors is a political scientist who has published good work in top journals in political science–ajps, jop, poq, etc.

second–Cosmides and Tooby are probably the most respect evolutionary psychologists alive. so, i’d be a bit more hesitant about just brushing them off as imbeciles.

third–psychology publishes studies like this b/c theres pressure to publish new, interesting, and non-obvious findings. which is why so much gets debunked. it’d be nice to see something new and non-obvious in political science–political cues affect behavior? shocking! the most interesting, non-obvious finding i can think of in political science (ppl do not vote their pocketbooks) comes form psychologists.

Andrew Gelman May 30, 2013 at 3:47 pm

Afewthings:

I haven’t seen anybody characterize the authors of this paper as imbeciles. But I do think their claims went way way beyond their data. Being respected evolutionary psychologists doesn’t mean they know much about politics. I’m a respected political scientist but I don’t know much about macroeconomics. Etc.

With regard to your last point, there’s a lot of data and theory in political science, not coming from psychologists, showing that people do not vote their pocketbooks.

yes May 30, 2013 at 4:29 pm

the pocketbook voting stuff comes psychologists–kinder and sears–not political scientists.

Andrew Gelman May 30, 2013 at 8:19 pm

Huh? Kinder and Sears are political scientists.

Mark May 31, 2013 at 2:03 am

Sears, at least, has joint appointments in both psych and PS departments.

Andrew Gelman May 31, 2013 at 6:46 am

Mark,

I have a huge respect for many psychologists and much of their research, including their research on political topics. The Petersen et al. paper, however, seemed to reveal naive views about political attitudes, and it seemed to me to be the sort of work that would not be publishable in a top political science journal.

Mark June 4, 2013 at 8:03 am

I know that this conversation is long over now, but I wasn’t being critical (I’m not “yes” or “afewthings” etc.). I was just noting that David Sears has joint appoints – he is both a psychologist and a political scientist.

Tom May 30, 2013 at 1:40 pm

You state, “The thing that bothers me about these various mini-papers that get so much attention is that they’re hyped beyond all recognition.” This statement is true due to the nature of the subject. However, what bothers me most is that academics who produce serious work can’t figure out how to promote their own work across the media successfully. It’s pathetic. As a result, I’m stuck having to read crap about weird studies like the ones you mentioned.

Wonks Anonymous May 30, 2013 at 3:30 pm

“I’d say that maybe we need a journal for speculations”
There used to be a journal called “Medical Hypotheses” that served that purpose, but then angry people badgered Elsevier into shutting it down because Duesberg was publishing in it.

This is what Eliezer Yudkowsky calls an “especially elegant evpsych experiment”.

Iohannes May 30, 2013 at 8:12 pm

But PLOS-One costs the author something like $1300 to publish. Hardly a place to “put some shaky empirical work paired with speculative conclusions.”

Iohannes May 30, 2013 at 8:15 pm

And Michael Petersen is a political scientist BTW.

Andrew Gelman May 30, 2013 at 8:17 pm

I took the press release at its word when Petersen was described as a “psychological scientist.”

Jonathan Baron June 4, 2013 at 8:00 pm

I finally got my e-copy of Psych. Science and saw the one about the upper-arm stuff. I wonder if this has anything to do with self-interest. Perhaps strong-arm tough guys just take more extreme positions on anything, even when it has nothing to do with self-interest.

Andrew Gelman June 4, 2013 at 9:25 pm

Jonathan:

Don’t forget that most of these data are on college students, and the controlling for age was done crudely (in a strong-arm way, one might say).

Comments on this entry are closed.

Previous post:

Next post: