A junior scholar, whom I’ll call Pat, writes with the following question:
bq. As a young researcher, I am conflicted about sharing working papers on my website or on SSRN. While I certainly understand the importance and possible utility of sharing work and possibly getting feedback, it also seems that there are dangers of posting pre-peer reviewed articles: having it publicly trashed prior to submission, losing the anonymity (kind of) guaranteed during the peer review process.
Since we often discuss working papers on this blog, I’ll hazard a few thoughts. First, one benefit of sharing, as Pat says, is additional feedback. But more than this, you also get the benefit of having your work circulate more widely. Some significant part of being (viewed as) “successful” in academia is being visible to your peers. Yes, the actual quality of your work matters more, but visibility counts for something. In tenure decisions, departments often want evidence that you are “known” in the field. There are lots of ways to build visibility — going to conferences, networking, etc. — but most all of them involve sharing your work, even in its early stages. Pat raises the possibility that your work may get trashed. That can happen, but I think positive or constructive feedback is more common.
Second, I think there are benefits to science from having working papers shared. Peer-reviewed journals are valuable for many reasons, but they create pathologies. A lot of deserving research is not published. (Acceptance rates at top political science journals are below 10%.) And not only that, but often it is certain kinds of research that are not published, such as research with null findings (leading to the well-known file drawer problem) and research that replicates an earlier study (which everyone agrees is valuable but few journals seem to want to publish). At least if working papers are publicly available, there is some chance that such research will achieve visibility even if it is difficult to publish. Moreover, there is also the extraordinary lag between submitting to a peer-reviewed journal and (if lucky) actually seeing the article in print. This can take 2 years or more — perhaps reason enough to circulate working papers. (For more thoughts on the value of having research circulate before peer review, see Paul Krugman.)
What are the problems of sharing working papers? Pat raises the possibility that it will compromise the anonymity of double-blind peer review. Of course, Pat qualifies this (“kind of”), which leads to my thought: given the existing ways in which peer review often isn’t blind — such as because papers have already circulated at conferences — I don’t think that sharing working papers on a website or SSRN has much additional effect. Moreover, given that acceptance rates at journals are already so low, I just don’t think it makes that much difference when a paper’s early circulation ends up making the resulting peer review process less-than-blind. In a world with acceptance rates of 8-10%, there’s just not much a scholar can do to “game” that outcome for good or ill (other than try to produce better work, in which case we’re back to the value of feedback on working papers).
To me, the more serious challenge with working papers is their possible negative consequences for science as a whole. The World Bank’s Berk Ozler had a good post about that a couple years ago. He points out that sometimes findings change between early versions and later versions of papers. But people’s interest in first version of the research often outstrips their interest in the revised version. Ozler:
bq. People are busy. Most of them had only read the abstract (and maybe the concluding section) of the first draft working paper to begin with. Worse, they had just relied on their favorite blogger to summarize it for them. But, guess what? Their favorite blogger has moved on and won’t be re-blogging on the new version of the working paper. Many won’t even know that there is a more recent version. The newer version, other than for a few dedicated followers of the topic or the author, will not be read by many. They will cling to their beliefs based on the first draft: first impressions matter. By the time your paper is published, it is a pretty good paper – your little masterpiece. The publication will cause an uptick in downloads, but still, for many, all they’ll remember is the sweatshirt, and not the sweat that went into the masterpiece.
And it could get even worse:
bq. There is another problem: people who are invested in a particular finding will find it easier to take away a message that confirms their prior beliefs from a working paper. They will happily accept the preliminary findings of the working paper and go on to cite it for a long time (believe me, well past the updated versions of the working paper and even the eventual journal publication). People who don’t buy the findings will also find it easy to dismiss them: the results are not peer-reviewed. At least, the peer-review process brings a degree of credibility to the whole process and makes it harder for people to summarily dismiss findings they don’t want to believe.
I don’t think these problems mean that Pat or any other specific person shouldn’t share working papers. They might think through the question “what is likely to change in this paper moving forward?” — and if they feel that the empirics are very solid, they might be more inclined to something publicly.
But the problems Olzer mentions are obviously broader, and demand a disciplinary response. But what? Ozler comes down on the side of speeding up peer review, thereby helping to ensure that any political or policy response to research takes place after peer review. That’s hard to do — ask any journal editor how easy it is to get peer reviewers (Ozler notes that some journals are actually paying reviewers) — but I support the idea of speeding up in principle.
Ultimately, I’d say that the potential benefits of sharing likely outweigh the costs for any individual researcher. For disciplines as a whole, the picture is murkier. Figuring out how to extract the good that comes from sharing working papers while avoiding the bad isn’t easy.
I welcome thoughts in comments.