A couple of days ago, I outlined my thoughts on two of a set of seven ideas that have been proposed for improving scholarly journals—that articles should be shorter and that more of their content should be laid off to Web sites. Although the merits of those two suggestions are obvious, I have some qualms about both of them.
Now I’ll turn to the third suggestion.
Editors should decline to send out a significant number of submitted articles for review. Any editor worth his or her salt recognizes an article at the time of submission with no probability or a low probability of success.
I suppose we could all agree with this suggestion in principle, but here as so often is the case the devil may lie in the details. What’s “a significant number”? 5%? 20%? 40%? When we move from principle to practice my “in principle” concurrence begins to wear a bit thin.
The premise underlying this suggestion is unquestionably correct. A large proportion – even, though I hesitate to say this publicly, more than half – of the papers that a journal – even a top-tier journal – receives are instantly recognizable as non-starters. In some instances, that’s because the submission is substantively inappropriate for the journal; no matter how broadly the journal’s scope is defined, the paper simply falls outside of it. In some other instances, it’s because the paper wasn’t ready to be sent out; not enough sweat has been invested in it and not enough pairs of eyes have seen it. At the APSR, we received many papers that were riddled with typos (even in the title), full of basic grammatical errors, and so on – in short, papers that hadn’t even been spell-checked, let alone carefully edited and re-edited. More commonly, though, we received papers that weren’t ready to be seen because the author hadn’t yet decided what the paper was about. The author may, for example, have run some regressions and provided a detailed report of the results without even attempting to define a research issue crisply or to make a case for its importance. At least at the APSR, though, the most common shortcoming was none of the above. Rather, it was just that the paper wasn’t “big” enough, in a theoretical and/or substantive sense, to warrant publication in a top journal. It may have been a competently executed analysis, but it really belonged to a subfield journal, given its incremental contribution.
So yes, lots of papers – many, even most – could be rejected by the editor after a quick scan. (Though I’d be less comfortable about having an editor reject a paper for the last-named reason than for the preceding ones, given that the paper may center on issues far removed from the editor’s own expertise.)
But let’s ask ourselves why a larger portion of papers should be rejected without being sent out for review, and why they shouldn’t be.
One reason to bypass the review process is to save time for the author. Who wants to sit and wait for month after month for a rejection letter, especially when it turns out that the paper wasn’t a bona fide contender in the first place? This is a valid concern, but it’s much less valid than it used to be, because (horror stories aside, and everyone has one) journal turnaround times are so much faster than they used to be. At the leading political science journals these days, the turnaround time for papers that go through the full review process tends to be two to three months. That’s not a long wait.
Another reason to bypass the review process is to save time and effort for reviewers. It’s not much fun to be sent a lousy paper and asked to review it, and reviewing lots and lots of papers (many of which are lousy) puts a burden on reviewers. By the way, one major reason why the burden on reviewers has gotten so heavy is that turnaround times have gotten so short; now an enterprising author can get a paper rejected by three or four different journals in a single year. On the other hand, when a reviewer complained to me about the poor quality of a paper I had sent for review, I would ask which type of paper requires more time and effort on the reviewer’s part: a good one that could be better, or a an irredeemably lousy one. Almost invariably the answer would be “the good one,” because it doesn’t require a close reading to determine, or a lengthy review to describe, what’s wrong with a lousy paper. Now assume that I, as editor, am going to send you no more than a certain number of papers per year to review – say, three or four. In terms of conserving your time and energy, I’m actually doing you a favor by sending you lousy papers.
Do you think I’m kidding about that last point? Well, to some extent I am, but only to a very limited extent. Consider the following: The best review I ever saw, of the thousands I read while editing the APSR, began as follows: “I love this paper.” Then the reviewer went on to present 22 single-spaced pages full of specific and constructive suggestions about the paper could be improved. The reviewer must have spent at least a week preparing that review. That’s obviously an extreme case, but it dramatically illustrates the point.
Along the same lines, I occasionally received complaints from reviewers to the effect that they were so overburdened with writing reviews that they didn’t have time to do their own work. In many instances, when the reviewer told me how many reviews he or she had completed recently, I sympathized and let him or her off the hook. But in many cases I didn’t. Here’s the way such conversations often went:
Reviewer: I’m way too busy writing reviews. This is a real burden.
Me: I’m sorry to hear that. How many reviews did you write last year?
Reviewer: Let’s see: Eight or ten, I guess. Maybe one every month. That’s terrible.
Me: And how many papers of your own did you submit for review?
Reviewer: I finished three new papers last year.
Me: And were they all accepted?
Reviewer: Well, no. In fact, all three were rejected.
Me: And did you then send them to another journal?
Reviewer: Yup.
Me: And were they then accepted?
Reviewer: Hmm. Two got revise-and resubmits, and one got rejected.
Me: And did you revise and resubmit the two and submit the other one to a different journal?
Reviewer: Yes.
Me: And did you have any older papers that were out for review last year?
Reviewer: Yah, two or three of them that I’d finished the year before and revised.
Me: Okay, now let’s do the math. Last year you reviewed eight or ten papers, maybe twelve. During the same year, you generated a need for, let’s do the math, three reviews each of your three new papers on the first time around – that’s nine reviews – plus three more apiece when you sent them out for the second time – that’s nine more reviews – plus three more apiece on the third round – that’s nine more reviews – plus, say, six to nine for the older papers you were still circulating from the previous year. So you wrote somewhere between eight and twelve reviews while creating the need for somewhere around 35 reviews. It sounds to me like you’re not carrying your weight. You’d better get busy writing some more reviews.
Reviewer: [Sputtering noise heard – expletive deleted.]
Now, two other points:
The top journals are overrun with submissions. It’s unrealistic to expect the editor of such a journal to provide anything beyond a cursory explanation of why a paper wasn’t even sent out for review. Authors, for better or worse, expect to have their work appraised in more than a cursory way – you do, I do, and so do they. Having your paper returned with a summary rejection is not at all a happy occasion, and tends to breed considerable ill will for the journal and to foster the sense that one hasn’t gotten a fair hearing. What do you, as an author, learn from having an editor tell you, in virtually so many words, that he or she isn’t going to open the review process on your paper because it doesn’t appear to be a strong candidate for publication?
And here’s the double-whammy part of that reservation: A disproportionate number of the lousy papers are submitted by junior scholars. For them, getting their work reviewed by those who are knowledgeable in their field can be an important socialization experience from which they learn a lot. By contrast, getting their work summarily dismissed by an editor gives them almost nothing to go on. The peer review process is supposed to produce better scholarship, in the long term if not always immediately; but that goal gets short-circuited if one doesn’t get one’s work reviewed.
Journals vary. Editors vary. Situations vary. In many instances, more papers should be rejected without being sent out for review. Prominent among those are papers in the first couple of categories I mentioned above: papers that clearly fall outside the scope of the journal and those that are so sloppy that they shouldn’t have been submitted. If a paper meets the minimum criteria of pertinence and care in preparation, though, I tend to believe that the advantages of sending it out for review outweigh the disadvantages.
{ 9 comments }
Lee, I like this series of posts so far. As a junior scholar (advanced grad student), I feel like I am learning a lot reading it. I wanted to write a brief comment to this post. The work that I do straddles two disciplines, sociology and public health. The sociology model tends to sound much like the one in political science (and psychology from the original inspiration for this series) and public health more like neuroscience. Public health has the quick summary review to decide whether to send it out for a lengthier review. Although it stings a little bit having my work outright rejected, the fact that I get that rejection within a week of submitting a paper to a public health journal makes the rejection go down a little easier. Although a review would have been nice, waiting three months to have three reviewers and an editor tell me that my paper didn’t have a snowball’s chance stings even more.
I have obviously never edited a journal (in fact, I have done very few reviews); but, if it is initially obvious that over half of submissions aren’t going to make it in a journal, it seems like a good rule of thumb would be to reject them outright if there is no chance of the paper making it. And, if there is any chance at all, err on the side of giving it a full review.
Perhaps the solution isn’t summary rejection by editors, but rather very brief R&Rs from editors:
“We noticed that you have not taken the time to proofread your paper thoroughly. Before we begin the review process, please take the time to edit and revise your paper more carefully…”
Or, in the case of a paper that seems too narrow for a top journal, send a note advising the author of that problem and asking whether they really want to spend three months on the review process, and strongly suggesting that they send it to another journal.
I like Adam’s suggestion a lot.
I agree that we need to reduce the burden on reviewers, and that editorial screening is a valid method. However, I think that journals would be well-served to craft much better summary rejection methods than simply editor capriciousness (which, in the absence of more solid comments, is what submitters will think).
I think that Adam’s phrasing might be a little off, but the gist is right.
But, by the same token, some of the editorial screening needs to be relaxed. I know senior scholars who had given up on submitting to the top journals because of the “don’t cite yourself” rule. For some senior scholars, I would reject their work for publication if it DIDN’T cite their earlier work. And, in the past, I have easily figured out who I’m reviewing by seeing the omitted references. Rather, editorial staffs might search submissions for truly incriminating lines like “as my previous work (2005) showed….” instead of a third person citation to a published (or even conference) work.
Lee,
I love the point on supply and demand, and I’ve been trying to make that point myself. I just don’t know the solution.
If each one of us looked at how many submissions we made last year and if we reviewed 3 times that number of articles, I’d think we’d discover that there is a serious disconnect.
I suspect the bulk of the supply comes from assistants and associates in “publish or perish” departments, who are sending out 6, 7, or more articles annually. Do those same colleagues review 20+ articles a year?
This is a really tough nut. For my own part, I have simply cracked under the demand in the past few years, particularly since I am now at a school where my publication rate just doesn’t matter much for advancement and promotion, so I am submitting at best 2-3 articles a year.
I feel some professional obligation, but not to perform two dozen reviews annually.
You make excellent points against having an editor summarily dismiss an article, outside of rare cases where the submission is clearly inappropriate (e.g., submitting a _Foreign Affairs_ type article to APSR or ISQ). But this is different from discouraging poor practices — it would be appropriate to send a typo-ridden article back to the author demanding that they proofread it before sending it out.
As for just poor papers, reviewers should feel free to reply in a single graf: “This paper is poorly written. The research aim is unclear, what is clear is marginal to research in this area, and the conclusions are murky.”
One of ISQ’s reviewers complained that he was tired of getting sub-par work from grad students to review and that the editors should reject it summarily (in fact, he had never been sent a grad student’s paper). One editor told him to write his review in a paragraph: “This is sub-par work that reads like a grad student looking for advice rather than a polished paper for publication.”
For this problem I propose a new category that reviewers can use: Reject without Review. Editors would request that referees use this only for papers that do not meet the standard for submission to a scholarly journal. The reviewer would determine what is that standard.
The most appealing part of this proposal is that it improves information. I believe that many reviewers take a quick peek at most of the manuscripts they are sent, and only after doing so decide whether to review them. If the ms. is obviously a stinker (one that deserves a Reject w/o Review) I conjecture that many reviewers ignore the request (or decline to review). Some journals have a less than 50% response rate on requests to review, and my belief suggests that these are skewed disproportionately toward stinkers that fall into Lee’s category of the 50% that should not have been submitted. And if my conjecture is on target, then reviewers are failing to pass on useful information to editors because they perceive the cost of doing so to be too high. So my proposal says: let’s lower the cost and make it easy to share that information. Every editor to whom I have spoken about this agrees that less than half of the articles they receive warrant a review. But they are reluctant to unilaterally make that decision. Let’s allow reviewers to make that call for them.
Perhaps I am mistaken about the cost of reviewing stinkers and Lee is correct: most people find it easy to write a review for a lousy paper (I do not, but I am often an outlier). This proposal will not cause a problem in such a world: since writing reviews of lousy papers is not very difficult, people will tend to do it.
Reviewing a bad paper takes less time, but it is time wasted. Reviewing a good paper takes more time, but it is time well spent. The net cost to the reviewer is far higher in the first case than the second.
It makes sense for editors to give assistant professors’ papers greater consideration — assuming they meet minimal standards of scholarly discourse — but otherwise editors should just say no.
Reviewing a bad paper takes less time, but it is time wasted. Reviewing a good paper takes more time, but it is time well spent. The net cost to the reviewer is far higher in the first case than the second.
It makes sense for editors to give assistant professors’ papers greater consideration — assuming they meet minimal standards of scholarly discourse — but otherwise editors should just say no.
Fred: I don’t agree that reviewing a bad paper is time wasted. If the review is to the point without being obnoxious, the author may well learn something from it and do better next time — not a bad outcome. By contrast, a more extensive review of an already-good paper is likely to produce at best a marginal improvement in it — not a bad outcome, either, but in many instances not one that warranted the time and effort.
Comments on this entry are closed.