Is Peer Review Broken?

by John Sides on May 24, 2013 · 4 comments

in Academia

A very discomfiting article (gated):

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.

With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.


[UPDATE: A commenter draws my attention to the publication date of this article: 1982.  That’s what happens when you learn about something from Twitter: you assume it just happened today.  Sorry.  But this raises an interesting question: would the same thing happen again if this study were done today?]

Sympathy for the Kinsley

by Andrew Gelman on May 24, 2013 · 6 comments

in Media,Political Economy

Paul Krugman, Daniel Drezner, and others slam fabled contrarian Michael Kinsley for his argument that we need to cut the budget deficit now because “we have to pay a price for past sins, and the longer we put it off, the higher the price will be” and that this attitude follows “the lessons of Paul Volcker and the Great Stagflation of the late 1970s.”

You know you have a problem with Drezner when, instead of calling your work “piss-poor monocausal social science,” he doesn’t even call it “social science” at all.

I have some sympathy for Kinsley, though, not on the merits—-I have no idea on the merits, for all I know he’s exactly correct on the economics—-but on his reactions to this event.

I think I’m well qualified to write about this because I know about as much of macroeconomics as Kinsley does (or so I suspect).
[click to continue…]

The following is the first in our series of collaborations with journals to feature guests posts from authors of recently published political science research in conjunction with ungated access to the article that is being discussed.

The guest post is written by political scientist Roland Paris, of the University of Ottawa (@rolandparis).  An ungated version of his article is being made available temporarily by Cambridge University Press here.

*****

In the June 2013 issue of Perspectives on Politics, I have a review essay based on four books that offer insights into “what went wrong” with the international effort to stabilize Afghanistan after 2001.

The books, which are all excellent, approach the subject from different vantage points.  Astri Suhrke’s When More Is Less: The International Project in Afghanistan examines the internal tensions and contradictions of the overall international effort.  Rajiv Chandrasekaran’s Little America: The War Within the War for Afghanistan focuses more narrowly on the US military and civilian “surge” in 2010 and 2011.  In Bazaar Politics: Power and Pottery in an Afghan Market Town, Noah Coburn conducts a micro-level analysis of the politics in one village near Kabul during the international mission.  Finally, Thomas Barfield’s Afghanistan: A Cultural and Political History is a macro-history of Afghan politics and governance from pre-modern times to the present.

In spite of their differences, all of the books point to similar, underlying dysfunctions in the international mission.  The first dysfunction was the interveners’ inadequate understanding and knowledge of Afghan society.  Again and again, the authors point to cases of international action rendered ineffectual or counterproductive due to a lack of familiarity with the political and social environment. From the highest levels of decision making to the micro-dynamics of military patrols and aid projects, foreign organizations and officials seemed to be almost handicapped by their own ignorance of the country.

The second dysfunction was the persistent short-termism of international policymaking.  At each major juncture, decision makers seemed to reach for the most expedient fixes without fully considering the context or consequences of their actions.  This pattern was already visible during the 2001 invasion, when the United States paid Afghan militias to intercept fleeing Al Qaeda fighters in the mountains of Tora Bora, rather than sending American forces—a costly decision, since the militias turned out to be less than fully committed to the task.  Then there was the Bush administration’s lack of interest in devising plans for Afghanistan’s post-Taliban transition, and its eagerness to delegate this task to others, based in part on the assumption that the “problem” of Afghanistan had been largely resolved by the defeat of the Taliban regime.  Next came the UN-sponsored conference at Bonn, which produced an agreement for a political transition process.  This agreement, however, was reached “hastily, by people who did not adequately represent all key constituencies in Afghanistan,” as Brahimi, who chaired the meeting, wrote in a contrite essay seven years later.  With US and UN backing, moreover, the Bonn plan yielded a highly centralized system of government that was ill-adapted to the country’s needs.  Meanwhile, Washington had rejected the idea of deploying ISAF outside Kabul and refused to allow US counterterrorist forces to be used for “nation-building” purposes.  All of these actions reflected wishful thinking— or, more precisely, a dearth of serious thinking—about the viability and long-term implications of these decisions.

This mind-set continued in subsequent years.  As conditions worsened and the scale and scope of the operation slowly expanded, there was little reflection on the underlying assumptions of the mission.  When the US government, long distracted by the situation in Iraq, shifted its attention back to Afghanistan in 2008, decision making became more urgent, but was no less short-sighted. “Again and again,” writes Suhrke, “it was hoped that the latest change in strategy and personnel or increase in aid would be the silver bullet.”  I saw this for myself during visits to Kandahar and Kabul in December 2008 and January 2010.  Activity was intense, almost frantic, and driven by a sense that little time remained to “turn the situation around.”  But exactly how this would be achieved, and to what end, were never clear.  Even after President Obama entered office and conducted a lengthy policy review that resulted in a sharp escalation of US forces, these questions remained largely unanswered:  How would the United States convince the insurgency to capitulate or negotiate?  How would it persuade Afghan villagers to side publicly with ISAF and the Kabul government?  What, in short, was the purpose of the surge?  More broadly, why did the international operation, with its minimalist start and late escalation, seem so strangely out of sync with conditions on the ground?

If “strategy” is a plan of action designed to achieve a long-term or overall aim, there appeared to be little strategy guiding the international operation in Afghanistan.  Instead, reliance on a series of quick fixes seemed to substitute for strategic thinking—or tactics without strategy.  I conclude the essay by discussing the dangers of repeating these errors and drawing the wrong lessons from the Afghanistan episode.

You can access the entire essay for free until June 23, 2013.

In line with both the mission statement of this blog and recent discussions about the continuing need to highlight to those outside the academy the importance of political science research, we are in the process of trying to offer a new service here at The Monkey Cage. The goal is to have posts from authors whose research is featured in just released issues of political science journals provide a short guest post highlighting the research in conjunction with an agreement from the press to make that article available ungated for a specified period of time. This will be similar to what I have done in the past with some of the section newsletters, but now will feature research articles from journals. And we have gotten journals in the past to ungate articles in conjunction with posts on The Monkey Cage (and will continue to do so), but the hope is that this new set up will help regularize process and make a bit less ad hoc.

This means that if you are a reader of the Monkey Cage and lack institutional access to political science journals, The Monkey Cage will now be a way to access some very recent journal publications. Moreover, our hope is that the guest posts will be written in a way that will quickly make it apparent whether the full article would be of interest to you. I will write more when I have details on which journals are participating, but I wanted to just make this brief announcement now because I will momentarily be posting the first article under this arrangement. Please be aware that the articles will be only ungated for a limited time, so this access will work better if you read the post in real-time as opposed to searching for them later, although I will use the category “Journal Collaboration” to tag these posts.

For authors who are requested by journal editors to participate in this program, I strongly urge you to consider doing so. It is a great chance to get publicity for your research, and also an opportunity to contribute to a public good of helping to create more links between published political science research and the wider community of journalists, policy makers, private sector actors, politicians, etc. who may be interested in the work that we have put so much effort into producing.

Comments from Rich Arenberg interpreting yesterday’s Senate floor dust-up between Reid and McConnell over changing Senate rules:

Maybe the confusing label “nuclear option” which has been given to potential procedural maneuvers in the Senate which could lead to a majority-only rewriting on the Senate rules is more apropos than we thought.

During the Cold War, U.S. and Soviet intelligence each analyzed even the most subtle moves of the other side. If Soviet subs moved a few miles closer to the U.S. shores, the Air Force might move its long-range bombers outside their normal hangers and park them on the tarmac. The subs would move farther away and then the planes would roll back into their hangars. Signals had been exchanged. Most of this was invisible to most people and at best, confusing.

Watching the Senate’s leaders execute the delicate dance which is so often a part of the Senate’s approach to difficult confrontations is similarly difficult to interpret.

Yesterday, Senate Republican leader Mitch McConnell and Majority Leader Harry Reid exchanged their own signals on the Senate floor. Understandably observers were confused and interpretations varied widely.

For example, Sahil Kapur writing in TPM declared, Reid “rebuffed” McConnell’s “warning not to follow through with his threats to weaken the filibuster for nominations via the nuclear option.” The headline of Greg Sargent’s analysis in Washington Post’s Plum Line declared, “Harry Reid escalates ‘nuclear’ threat.’”

These interpretations were based on a Reid statement off the Senate floor. The majority leader said, “Despite the agreement we reached in January, Republican obstruction on nominees continues unabated. I want to make the Senate work again – that is my commitment.”

Others, like Tom Curry in NBC News First Read read the signals entirely differently. His piece appeared under the headline, “Reid appears to back away from ‘nuclear option’ on filibusters.”

That analysis, which I also heard from other sophisticated insiders in Washington, was based on Reid’s statement on the Senate floor: I am not saying we are going to change the rules, but I am saying we have to do a better job than what is going on around here. This is no threat.”

Clear right?


 

The senior senator from Arizona urged this body to trust the Republicans. Let me be clear, I don’t trust the Republicans. I don’t trust the Democrats and I think a whole lot of Americans likewise don’t trust the Republicans or the Democrats because it is leadership in both parties that has got us into this mess.

So said Ted Cruz today.  This is not the first time he’s run afoul not only of Democrats—Harry Reid called him a “schoolyard bully” a few weeks ago—but also of his fellow Republicans, conservative allies (e.g., the Wall Street Journal editorial board), and others ostensibly on his side.  You can follow the links to various stories in this Ramesh Ponnuru piece.

Ponnuru, who has known Cruz for a long time, thinks his behavior is just fine.  Confronted Cruz’s violations of Senate norms, Ponnuru asks, in essence, “so what?”

The people of Texas didn’t vote for him because he promised to keep his head down in deference to his colleagues. No senator wins election that way. Presumably voters want senators who will be as effective as they can be in advocating for the views they campaigned on.

Maybe that’s a correct theory of the voting behavior of Texans, and maybe not.  Regardless, I don’t think Cruz’s problem is in Texas.

Cruz’s problem is that he may want to be president of the United States, reports the National Review’s Robert Costa.  And to be the Republican nominee, he’ll need the support of his Republican colleagues.  The 2012 election once again showed—and despite some skepticism—that it is very hard to win the nomination unless you’re preferred by a substantial chunk, if not the vast majority of, your party’s leaders (as was Romney).  Which is to say, it pays to be nice to your colleagues.  It’s no guarantee, of course: junior Senator Hillary Clinton kept her head down and played nice, and lost the nomination.  John McCain often irritated his fellow Republicans, but still mustered enough support within the party to win the nomination.

But nevertheless, Cruz’s strategy is risky.  Just ask Santorum or Gingrich, whose surges in the 2012 primary were met by deafening silence from GOP leaders (Santorum) or vigorous, on-the-record denunciations (Gingrich).

Ponnuru, for his part, acknowledges something like this:

Cruz’s Beltway critics were horrified anew last week on reports that he is thinking of running for president. If the past few months are any guide, he would try to build a majority starting with the most conservative end of the Republican primary electorate, and argue that the party needs to nominate a true conservative rather than an establishment favorite.

Many Republicans have tried this strategy since President Ronald Reagan left the scene. None has succeeded. The right end of the party invariably splits its support among several candidates, and voters in the middle of the party usually prefer someone with more experience than the right’s favorites.

Maybe Cruz has a different strategy in mind, or has a reason for thinking this time will be different. If he runs, he will have at least one thing going for him: He has a knack for making his opponents lose their wits.


Yes, “none has succeeded.” That’s a warning sign right there.  And the other warning sign is the reason why they haven’t succeeded, which is not quite what Ponnuru states.  It’s not that the conservatives can’t coalesce on a candidate and the middle prefers someone with “experience.”  It’s that many in the party are wary of nominating a strongly ideological candidate—a “severe” conservative, if you will—because they’re afraid that this candidate will lose in the general. And rightly so!  See Table 2 of this article (pdf) by John Zaller.

In 2012, as Lynn Vavreck and I show in The Gamble (pdf), only about half of Romney’s supporters were closest to him ideologically. The other half were closer to Gingrich or Santorum—that is, to one of the more conservative candidates.  But this other half also tended to believe that Romney was the only candidate who could beat Obama in November.  Their vote was about electability, not experience. Of course, Romney didn’t win, but Zaller’s finding suggests that a more conservative candidate would have done worse, other things equal.

Cruz’s path to the presidency—if he decides to run—must consist precisely of convincing “the middle” of the party that he’s electable despite the fact that he may be the most conservative member of the Senate (pdf).  To do that, he’ll need the support of his fellow party leaders to send that signal.  It doesn’t matter if he has “a knack for making his opponents lose their wits.”  His opponents will be busy nominating Hillary Clinton or whoever.  And it doesn’t matter whether, deep in his heart, he trusts Republicans.

What matters is whether he has a knack for making his fellow Republicans trust him.

[Photo credit: Gage Skidmore]

The Sociology of Think-Tanks

by Henry Farrell on May 22, 2013 · 3 comments

in Blogs

This article in The Nation is likely to cause some considerable embarrassment to the Center for American Progress (CAP).

Though the think tank didn’t disclose it, First Solar belonged to CAP’s Business Alliance, a secret group of corporate donors, according to internal lists obtained by The Nation. Meanwhile, José Villarreal—a consultant at the power-
house law and lobbying firm Akin Gump, who “provides strategic counseling on a range of legal and policy issues” for 
corporations—was on First Solar’s board until April 2012 while also sitting on the board of CAP, where he remains a member, according to the group’s latest tax filing. … Nowadays, many Washington think tanks effectively serve as unregistered lobbyists for corporate donors, and companies strategically contribute to them just as they hire a PR or lobby shop or make campaign donations. And unlike lobbyists and elected officials, think tanks are not subject to financial disclosure requirements, so they reveal their donors only if they choose to.
[click to continue…]

Special Prosecutors: Be Careful What You Wish For

by Andrew Rudalevige on May 22, 2013 · 0 comments

in Law,Presidency

There have been various calls in recent days for the appointment of an independent counsel to explore and expunge the IRS’s selective investigation of conservative groups seeking tax-exempt status after 2009.

From the right, Peggy Noonan in the Wall Street Journal claims, for instance,that “we are in the midst of the worst Washington scandal since Watergate.” If you can get past this cheap hyperbole (Ms. Noonan herself worked in a White House marked by a worse scandal, Iran-contra, and in 1998 she thought President Clinton deserved impeachment), her point is worth noting:

“Independent counsels should not often come in and distract the U.S. government from its essential business…. [but] what happened at the IRS is the government’s essential business. The IRS case deserves and calls out for an independent counsel, fully armed with all that position’s powers. Only then will stables that badly need to be cleaned, be cleaned.”


From the left, Bill Keller in the New York Times agrees: such an appointment would show the president truly takes any IRS wrongdoing seriously, he says, and “it’s the surest way to get answers the public might trust.” But most crucially, and here he intersects with Noonan, “The third reason for a special counsel is that the government has serious business to conduct, and the scandal circus on Capitol Hill is a terrible distraction.”

This all sounds very high-minded (Noonan even denounces as “shameful and shallow” any effort by “any Republican operative or operator to…turn it into a mere partisan arguing point and part of the game. It’s not part of the game. This is not about the usual partisan slugfest.”  Keller one-ups her by calling for the appointment of Clinton nemesis Ken Starr to the post.)

But if we know anything about the history of special counsel investigations, it is that we know that they are not, in fact, either high-minded or apolitical. The investigation may be nonpartisan, but the effects are not. The sideshow saga surrounding the investigation of the “outing” of Valerie Plame during the Bush II years is sufficient to give one pause on this point; but ponder, too, of course, the investigations by Starr or Lawrence Walsh or the five other independent counsels at work during the Clinton years. (Wake Forest political scientist Katy Harriger’s book on the topic gives much more detail.)

In short, appointing a special prosecutor may be the right thing to do. But the idea that this would partition off the IRS scandal from the rest of Washington and allow some sort of ‘space for governance’ is very dubious indeed.

To be sure, the structuring of any given special investigation could differ; but Noonan and Keller both call for a broad remit and total independence. Along these lines it is worth remembering the 1988 Supreme Court decision Morrison v Olson. By a 7-1 vote (Justice Kennedy did not participate) the Supreme Court upheld the constitutionality of the Independent Counsel Act (ICA), part of the Ethics in Government Act, a post-Watergate reform package passed in 1978. The ICA allowed for an odd quasi-judicial/executive appointment power for independent counsels, and IC target Ted Olson (yes, that Ted Olson) argued this structure violated the separation of powers by buffering counsels from the presidential removal power (and also wandered into Article III by the creation of a separate court). The Court held these were permissible infringements given the broader governmental interest in promoting the public trust and fighting executive branch corruption.

Only Justice Scalia dissented from this finding, a dissent which provided both a full-throated defense of “unitary executive” theory” and, more crucially for present purposes, a prescient vision of the use of the ICA as political weapon:

“The context of this statute is acrid with the smell of threatened impeachment…. [B]y the application of this statute in the present case, Congress has effectively compelled a criminal investigation of a high-level appointee of the President in connection with his actions arising out of a bitter power dispute between the President and the Legislative Branch. Mr. Olson may or may not be guilty of a crime; we do not know. But we do know that the investigation of him has been commenced, not necessarily because the President or his authorized subordinates believe it is in the interest of the United States, in the sense that it warrants the diversion of resources from other efforts and is worth the cost in money and in possible damage to other governmental interests….


“How much easier it is for Congress, instead of accepting the political damage attendant to the commencement of impeachment proceedings against the President on trivial grounds—or, for that matter, how easy it is for one of the President’s political foes outside of Congress—simply to trigger a debilitating criminal investigation of the Chief Executive under this law…. Besides weakening the Presidency by reducing the zeal of his staff, it must also be obvious that the institution of the independent counsel enfeebles him more directly in his constant confrontations with Congress, by eroding his public support. Nothing is so politically effective as the ability to charge that one’s opponent and his associates are not merely wrongheaded, naive, ineffective, but, in all probability, ‘crooks.’ And nothing so effectively gives an appearance of validity to such charges as a Justice Department investigation and, even better, prosecution….”


The brief filed in the Morrison case by three ex-Attorneys General, from the Ford, Carter, and Reagan administrations (Edward Levi, Griffin Bell, William French Smith) also seems apropos:

“[T]he institutional environment of the Independent Counsel—specifically, her isolation from the Executive Branch and the internal checks and balances it supplies—is designed to heighten, not to check, all of the occupational hazards of the dedicated prosecutor; the danger of too narrow a focus, of the loss of perspective, of preoccupation with the pursuit of one alleged suspect to the exclusion of other interests.”


Or, as Ted Olson himself put it, “If you are given a fishing license which has the name of a fish on it, and you don’t come back with that fish, you’ve failed.”

Certainly, in the present case, if there are criminal issues they should be investigated and charges brought. But simply calling someone a “special prosecutor” does not in itself cordon off a ‘safe space’ outside of which normal policy-solving political bargaining (if that’s normal now) can continue unaffected.  There are good reasons that the ICA was allowed to expire unceremoniously, with the blessing of both parties, in 1999.

thegamble_cover_small


Lynn Vavreck and I are pleased to announce the next e-chapter of our book on the 2012 presidential election, The Gamble.  The chapter, “High Rollers,” is here (with a pdf link here) and on Amazon.

This chapter is the first of three chapters on the general election campaign.  It deals with the summer campaign, including the continuing importance of fundamentals like the economy and party identification, the candidates’ messaging strategies, Obama’s early advertising blitz, the attacks on and news coverage of Romney’s tenure at Bain Capital, various “gaffes” (“You didn’t build that,” Romney’s foreign trip, etc.), and the Paul Ryan pick—basically everything up until the conventions.  You can see parts of that analysis previewed in my most recent post at Wonkblog.

Some findings from this chapter:

  • Economic conditions in the first half of 2012 still forecast an Obama victory.  Indeed, by at least one economic index, they were more favorable for Obama than for Clinton in 1996.
  • Despite concerns about “divided” political parties, Democrats and Republicans coalesced very quickly around Obama and Romney, respectively.  In general, party identification helped to create a great deal of stability in candidate preferences.
  • The various gaffes, as well as the attacks on Romney’s time at Bain Capital, produced at best small and temporary shifts in opinions.
  • News coverage of Obama and Romney during the summer was remarkably balanced.  The media favored neither candidate.  See the chapter for the data that support this claim.
  • The effects of advertising on vote intentions were visible but short-lived.  As I said at Wonkblog, we found no evidence that the early Obama advertising created durable shifts in opinion that would have benefited him on Election Day.
  • In the Pollster averages, Obama’s lead over Romney the day before the Republican convention began was exactly the same as it was on May 1.

We argue that the much of the general election campaign, especially in the summer, resembled a “dynamic equilibrium”:

The general election campaign resembles a concept from the sciences called a “dynamic equilibrium.” In a dynamic equilibrium, things are happening, sometimes vigorously or rapidly, but they produce reactions that are roughly the same size or magnitude and that occur at roughly the same rates. Thus, the entire “system”—populated by candidates, media, and voters—appears stable, or at a “steady state,” to use more scientific nomenclature. Reams of news coverage and vigorous campaigning coincide with stable polls.

The implication, then, is not that the Obama and Romney campaigns were ineffective.  It is that they were roughly equally effective.

The entire book is finished and will be available by late August or early September.  Stay tuned.

Here it is, courtesy of Boris Shor and Nolan McCarty.  Boris’s blog post about the data is here.  This is the first extensive data on the political views of state legislators.  Well worth digging into!