The following is a guest post by political scientist Scott Gehlbach (@sgehlbach) of the University of Wisconsin–Madison.
*****
The peer-review process, if not broken, is seriously under strain. Editors are forced to make hasty decisions based on imperfect signals from referees. Referees, in turn, are overburdened with review requests. And authors are at the mercy of referees who are not always qualified to evaluate all parts of a submission.
These three problems have a common cause: as a discipline we are asking referees to do too much. The typical review request takes the form: “Please evaluate this submission as a possible contribution to…” The referee process could be improved by adding a sentence that says, “As an expert in X, your thoughts on Y would be especially valuable,” where Y could be research design, a formal model, country context, framing, or any other element of the paper. In principle, the request could specify Y1, Y2, etc., though the point would be for the list to be less than exhaustive.
As an analogy, think of dissertation-committee members, who typically concentrate on parts of the dissertation where they have particular expertise. We don’t expect the Africanist to offer extensive comments on the model, or the formal theorist to advise on the ethnography, unless these specialists happen to be one and the same person.
Focusing reviewer effort in this way will allow editors to make better decisions, as it will be easier to extract the signal from the noise in referee reports. It will take a bit more work from editors at the front end of the review process, but I expect that it will save time at the back end. And I’m not convinced that it will take that much more time ex ante: editors already choose referees based on expertise (so, in the formulation above, they know X), and the increased use of desk rejection means that editors give manuscripts at least a cursory read on initial submission (so it shouldn’t be too difficult to fill in Y).
There are also benefits to other stakeholders. Referees will appreciate the sanction to direct effort to parts of a manuscript where they have the most expertise. And to the extent that all of this produces better and speedier decisions and more focused referee reports, authors (and ultimately readers) should profit.
A final note: There is a decentralized version of this reform that can provide many of the same benefits. Even if editors choose not to suggest that referees focus on particular elements of a submission, reviewers can still choose to restrict their comments in a way that reflects their substantive or methodological expertise. Indeed, I suspect that some referees do this already, but the key is to make it explicit: “In reviewing this manuscript, I primarily restrict my attention to Y.” Such a statement clarifies to editors and authors what the referee has, and has not, taken responsibility for.
I understand that there might be strategic considerations at play here. I leave those as an exercise for the comments section.