An Idea to Reduce Referee Burden

James Fowler posted an idea to reduce the burden of referees on the PolMeth listserv that is generating a lot of responses. We reprint it here, with his permission.

****

I frequently advocate for increasing the fraction of papers that get desk-rejected without review.

This not only reduces the number of referees that editors need, but it also reduces the cycle time from initial writing to publication for papers that would be a better fit at a different journal.

I notice that the fraction of desk-rejects is already creeping up at our top journals in political science, but it is still much lower than the rate in general science or natural science journals.

My colleagues who worry about increasing the fraction of desk-rejects note that one issue is that some authors will perceive the practice as being unfair—we need to give everyone an equal shot at review and we need to ensure that scholarship does not come to be dominated by a small number of players at specific departments.  Those are reasonable concerns.

So why don’t we do this instead: give authors the ability to opt-in to a higher desk-reject threshold (say, 50% get rejected without review).  In other words, an author can ask an editor to reject the paper quickly if he or she does not think it will succeed.

I am considering doing this voluntarily, and anyone else who wants to could as well.

But I wonder if it would help if editors explicitly gave authors the choice of two tracks for review, the current track, and a “fast track” with a higher desk-rejection rate.

This would preserve the potentially “fairer” set-up of the current track, while allowing authors who are interested in reducing cycle time to voluntarily give editors more power.

If enough authors opted in to the fast track, then the reduction in the burden on referees could be substantial.

And, interestingly, the choice of track by the author might send an informative signal to the editor.

Your thoughts?

4 Responses to An Idea to Reduce Referee Burden

  1. J. Otto Pohl June 8, 2011 at 10:58 am #

    Either you make the editor in charge of all acceptances and rejections and have no peer review or you have a peer review system. Giving the editor the ability to reject papers that are legitimate attempts at scholarship and then having a peer review process on top of it is grossly unfair and defeats a big part of the reason for having peer review in the first place. Of course editors often fudge the peer review system by sending out articles to reveiwers they know will reject a given article for ideological reasons. But, the original idea behind double blind peer review was to make the process as fair as possible.

  2. Ted Brader June 8, 2011 at 11:28 am #

    Thanks for bringing this important discussion to the Monkey Cage. I posted the following reply to James this morning. I’m not sure if it will be posted to the PolMeth listserv, but I thought I’d share it with Monkey Cage readers as well.

    ———-

    In regards to [James’] suggestion, I wonder if a two-track system would generate enough opt-ins to be helpful. Of course, one can always try and see. One potential implication seems to be that this track would likely be chosen much more frequently by younger scholars — ABDS, post-docs, assistant professors — who are the ones with the greatest incentives to want quick answers. That could be precisely the virtue of your proposal, or perhaps there are negative repercussions. I’m not sure myself at the moment, but I think it’s something to think through.

    As for desk rejects more generally, I think this is a reasonable practice, but recent applications that have come to my attention have raised a couple of red flags about the way desk-rejects might be used. First, journal editors need to make sure they at least give the paper a close enough look to ensure that their bases for a decision to desk-reject (or not) are accurate. In one recent case, for example, where I was an author on a paper, an editorial team’s comments made clear that, in their well-meaning haste, I assume, they were mistaken about the facts of what we had done in way that would obviously have big implications. Despite this, I felt the desk-reject process had likely been helpful to us overall on other fit and probability-of-success grounds for that particular journal. The factual misinterpretation, however, was a warning about the process generally, because it led to a misinterpretation of sufficient magnitude to imperil any paper at any journal were it the primary reason for a desk-reject.

    As individual reviewers, of course, we can all make mistakes, but we know a panel of reviewers is less likely to do so. Thus, the standard of “care” should go up when you revert to a single decision maker process.

    Second, and perhaps of even greater concern, I believe journal editors need to take care about applying overly narrow standards at a desk-reject stage for the valuable yet often contestable “importance of the contribution” criterion. In a recent example involving a paper by colleagues in the field (on which I was not an author), an editorial team made a desk-reject decision on importance-of-contribution that clearly favored certain types of contributions (e.g., methodological innovation, entirely novel theory) over other types of contributions (e.g., application a theory to an entirely new domain (relevant to the journal), unique original data that provides evidence on current debates of clear substantive importance in politics). This particular “judgment call” seemed to me so clearly a matter of narrow perspective or personal taste that I lost some measure of respect for the decision editor upon learning of the case (surely to be remedied, no doubt, the next time she or he accepts one of my own papers!). Authors (and their sympathizers) of course may always feel temptation to see things this way, which returns us to the issue of perceptions of fairness that [James] invoke[s] as a reason for the two-track process.

    But regardless of whether a two-track process is adopted, I believe editors have an obligation to take a broad and careful view on the importance-of-contribution criterion. We all know that particular criterion is simultaneously necessary, an “easy call” in many cases, and a subjective judgment large enough to import a boatload of bias (unconsciously or otherwise). Again, I certainly believe editors can make these sorts of importance judgments appropriately for many papers (e.g., when the papers make no more than incremental contributions on any dimension in a way that might be deemed inappropriate for a broad field journal) as well as make desk-reject decisions based on numerous other criteria — journal fit, substandard writing quality, substandard methods, etc. Nonetheless, in deciding whether and how to incorporate desk-rejects into the process, I urge editors (and organizations choosing editorial teams) to establish clear and cautious standards on the use of the importance-of-contribution criterion at that stage.

  3. Greg Weeks June 8, 2011 at 1:02 pm #

    I like the idea of improving the reviewing process, but I am not convinced that there is a crisis of reviewers being overburdened. Is there evidence of this? I tend to think there is a good-sized pool of qualified but undertapped professors (and even advanced Ph.D. students) and a smaller overburdened group of people who get tapped too much.

  4. Manoel Galdino June 8, 2011 at 5:06 pm #

    Even if there isn’t such a thing as overburdened reviewers, the system has flaws, some of them poitned by Zak Taylor in the polmeth list.

    The most important flaw, I think, is that we are commited to a pre-publication (peer review) filter which is a waste of time and resources. The reviews, nowadays, are not seen by anyone but the editor and authors. Readers can’t evaluate by themselves the claims of reviewers, even though is (potentially) an important output on their own. Moreover, they slow publication process and produce bias that no one knows how affects the final output, since it is not transparent.

    All in all, I think a radical change is needed. A few leading scholars in other fields, like Cameron Neylon [1], are arguing for a new system, which still has to be properly designed, of course, but is much needed.

    [1] http://cameronneylon.net/blog/reforming-peer-review-what-are-the-practical-steps/