Annals of Interesting Peer Review Decisions

Tom Bartlett describes the efforts of two psychologists to publish replication results for an article, which had purported to show that people could use ESP to predict whether they would be shown erotic pictures in the future. The replication found no observable effect, but (according to the authors’ account of it)had a difficult time finding a publisher.

Here’s the story: we sent the paper to the journal that Bem published his paper in, and they said ‘no, we don’t ever accept straight replication attempts’. We then tried another couple of journals, who said the same thing. We then sent it to the British Journal of Psychology, who sent it out for review. For whatever reason (and they have apologised, to their credit), it was quite badly delayed in their review process, and they took many months to get back to us.
When they did get back to us, there were two reviews, one very positive, urging publication, and one quite negative. This latter review didn’t find any problems in our methodology or writeup itself, but suggested that, since the three of us (Richard Wiseman, Chris French and I) are all skeptical of ESP, we might have unconsciously influenced the results using our own psychic powers. … Anyway, the BJP editor agreed with the second reviewer, and said that he’d only accept our paper if we ran a fourth experiment where we got a believer to run all the participants, to control for these experimenter effects. We thought that was a bit silly, and said that to the editor, but he didn’t change his mind. We don’t think doing another replication with a believer at the helm is the right thing to do … [the] experimental paradigms were designed so that most of the work is done by a computer and the experimenter has very little to do (this was explicitly because of his concerns about possible experimenter effects).

Although the Bartlett piece doesn’t make this suggestion, I can’t help wondering whether the reviewer was one of the authors of the original piece. Myself, I’ve had a couple of interesting interactions with editors over the years, but nothing that even comes close to matching this.

5 Responses to Annals of Interesting Peer Review Decisions

  1. Kat Clark December 7, 2011 at 11:48 am #

    The interesting implication of the BJP comments is that “psychic powers” or the belief of the experimenter could influence the results. Given the experimenters followed standard social psychology procedure (e.g. the computer doing most of the work), and they clearly are agreeing with the original Bem conclusions, doesn’t it follow that most of Social Psychology could therefore be up for critique on the grounds that most experimenters do have a hypothesis in mind before running a series of experiments. I mean, isn’t that why it’s called hypothesis testing?

  2. Kat Clark December 7, 2011 at 11:49 am #

    Argh. My vague non-referential “them” in the 2nd sentence are the reviewers and editor of BJP.

  3. PLW December 7, 2011 at 12:33 pm #

    This story makes me sad. The peer review process is a joke.

  4. Andrew Gelman December 7, 2011 at 7:52 pm #

    Setting aside the whole “psychic powers” thing, it makes sense to me not to run the new experiment. After all, it’s hardly news that ESP doesn’t work. If “ESP doesn’t work” were publishable, you could fill up a journal many times over with such findings. And what would be the point of that? Better to start a new journal with some catchy title such as Replications of Well-Known Findings. In the physics division, you could have articles demonstrating that objects fall down, not up. In the chemistry division, you could publish demonstrations that H2 + O2 yields H2O plus energy. The biology section could have a paper demonstrating that cats and dogs can’t produce offspring. And so on.

  5. Mark B. December 7, 2011 at 8:43 pm #

    It’s worth noting that at a conference I saw Bem talk about replication attempts of his now famous psi experiments. If my memory serves me well, a set of skeptics have run replication attempts and failed. A set of believers have run replication attempts and succeeded. So there might be something going on here (not psychic, but probably unconscious).

    Of course if the issue is the experimenter’s belief, then it invalidates the original findings as well.