The Promise and Perils of Sharing Work-in-Progress

by John Sides on September 9, 2013 · 5 comments

in Academia

A junior scholar, whom I’ll call Pat, writes with the following question:

As a young researcher, I am conflicted about sharing working papers on my website or on SSRN. While I certainly understand the importance and possible utility of sharing work and possibly getting feedback, it also seems that there are dangers of posting pre-peer reviewed articles: having it publicly trashed prior to submission, losing the anonymity (kind of) guaranteed during the peer review process.

Since we often discuss working papers on this blog, I’ll hazard a few thoughts.  First, one benefit of sharing, as Pat says, is additional feedback.  But more than this, you also get the benefit of having your work circulate more widely.  Some significant part of being (viewed as) “successful” in academia is being visible to your peers.  Yes, the actual quality of your work matters more, but visibility counts for something.  In tenure decisions, departments often want evidence that you are “known” in the field.  There are lots of ways to build visibility—going to conferences, networking, etc.—but most all of them involve sharing your work, even in its early stages.  Pat raises the possibility that  your work may get trashed.  That can happen, but I think positive or constructive feedback is more common.

Second, I think there are benefits to science from having working papers shared.  Peer-reviewed journals are valuable for many reasons, but they create pathologies.  A lot of deserving research is not published.  (Acceptance rates at top political science journals are below 10%.)  And not only that, but often it is certain kinds of research that are not published, such as research with null findings (leading to the well-known file drawer problem) and research that replicates an earlier study (which everyone agrees is valuable but few journals seem to want to publish).  At least if working papers are publicly available, there is some chance that such research will achieve visibility even if it is difficult to publish.  Moreover, there is also the extraordinary lag between submitting to a peer-reviewed journal and (if lucky) actually seeing the article in print.  This can take 2 years or more—perhaps reason enough to circulate working papers.  (For more thoughts on the value of having research circulate before peer review, see Paul Krugman.)

What are the problems of sharing working papers?  Pat raises the possibility that it will compromise the anonymity of double-blind peer review.  Of course, Pat qualifies this (“kind of”), which leads to my thought: given the existing ways in which peer review often isn’t blind—such as because papers have already circulated at conferences—I don’t think that sharing working papers on a website or SSRN has much additional effect.  Moreover, given that acceptance rates at journals are already so low, I just don’t think it makes that much difference when a paper’s early circulation ends up making the resulting peer review process less-than-blind.  In a world with acceptance rates of 8-10%, there’s just not much a scholar can do to “game” that outcome for good or ill (other than try to produce better work, in which case we’re back to the value of feedback on working papers).

To me, the more serious challenge with working papers is their possible negative consequences for science as a whole.  The World Bank’s Berk Ozler had a good post about that a couple years ago.  He points out that sometimes findings change between early versions and later versions of papers.  But people’s interest in first version of the research often outstrips their interest in the revised version.  Ozler:

People are busy. Most of them had only read the abstract (and maybe the concluding section) of the first draft working paper to begin with. Worse, they had just relied on their favorite blogger to summarize it for them. But, guess what? Their favorite blogger has moved on and won’t be re-blogging on the new version of the working paper. Many won’t even know that there is a more recent version. The newer version, other than for a few dedicated followers of the topic or the author, will not be read by many. They will cling to their beliefs based on the first draft: first impressions matter. By the time your paper is published, it is a pretty good paper – your little masterpiece. The publication will cause an uptick in downloads, but still, for many, all they’ll remember is the sweatshirt, and not the sweat that went into the masterpiece.

And it could get even worse:

There is another problem: people who are invested in a particular finding will find it easier to take away a message that confirms their prior beliefs from a working paper. They will happily accept the preliminary findings of the working paper and go on to cite it for a long time (believe me, well past the updated versions of the working paper and even the eventual journal publication). People who don’t buy the findings will also find it easy to dismiss them: the results are not peer-reviewed. At least, the peer-review process brings a degree of credibility to the whole process and makes it harder for people to summarily dismiss findings they don’t want to believe.

I don’t think these problems mean that Pat or any other specific person shouldn’t share working papers.  They might think through the question “what is likely to change in this paper moving forward?”—and if they feel that the empirics are very solid, they might be more inclined to something publicly.

But the problems Olzer mentions are obviously broader, and demand a disciplinary response.  But what?  Ozler comes down on the side of speeding up peer review, thereby helping to ensure that any political or policy response to research takes place after peer review.  That’s hard to do—ask any journal editor how easy it is to get peer reviewers (Ozler notes that some journals are actually paying reviewers)—but I support the idea of speeding up in principle.

Ultimately, I’d say that the potential benefits of sharing likely outweigh the costs for any individual researcher.  For disciplines as a whole, the picture is murkier.  Figuring out how to extract the good that comes from sharing working papers while avoiding the bad isn’t easy.

I welcome thoughts in comments.

{ 5 comments }

Jon M September 9, 2013 at 9:08 am

I think one additional concern people have is that pre-publication risks damaging the novelty of results. Let’s say I put up a working paper that outlines a new way of analysing a type of data.

If I’m unlucky with journal decisions there might be two or three papers using the ideas the paper discussed (with citations to the working paper). By the time the paper is published it’s possible that the reviewers may think that the article is not a very novel contribution to the literature (because the approach is already summarized in two or three other papers).

A related concern is that if you put a very rough piece of work up there is always the possibility that someone else will cite the rough work and write up a polished piece of work that supersedes the piece you would have eventually written.

I don’t think this is a knock down argument but there are definitely personal incentives for not putting undeveloped ideas up before you have a chance to work through their implications.

Disclosure: I regularly post my papers as working versions.

Jon M September 9, 2013 at 9:59 am

So actually I think the trade off is probablyore in the other direction. The discipline benefits from early access to results but researchers risk giving up the chance to fully exploit private information before making it public.

Jeff Bezos September 9, 2013 at 11:13 am

Yet another great post of the type that won’t appear once this blog moves to the Washington Post. Alas.

D. Mutz September 9, 2013 at 3:06 pm

I find it really interesting that you used the name “Pat” in your example, the quintessential androgynous name. Although it is certainly true that not all reviews these days are anonymous, most of what I review I cannot identify for certain.

Moreover, I know more than a few female scholars who always change the names of their papers between conference presentation and submission for publication, just so it won’t be too easy for the reviewers to find out whose paper it is by simply googling the title. We know biases exist, so why make it easy? Scholars are busy people, so if it takes any more effort than that, they’re not likely to bother trying to find out a paper author’s identity.

And as the comments suggest, most people don’t get past the title and abstract in looking over many working papers, so it’s not necessarily the case that reviewers will immediately recognize a paper they’ve glanced at before.

It would be interesting to know if female scholars are less likely to post working papers than male scholars. It would not surprise me, but I couldn’t say for sure.

Personally, I’ve never received constructive comments from posting a working paper, though the press certainly notices some of them, so I can see the merit of the visibility argument.

John Sides September 9, 2013 at 7:41 pm

D. Mutz: Yes, I chose “Pat” deliberately so as not to suggest whether the person who wrote me was male or female. The point you raise about whether female scholars post working papers at a lower rate than male scholars is very interesting. I couldn’t say for sure either. But certainly any gender biases in the review process deserve consideration when considering whether/how to share works-in-progress. Thanks for pointing out a missing element in my post.

Comments on this entry are closed.

Previous post:

Next post: