Via email, political scientist Jeffrey Lax responds to the Johnson, Wahlbeck, and Spriggs article I cited in my previous post. This discussion is only getting more relevant in light of reaction to the Solicitor General’s argument this morning.
*****
I think this is a nicely provocative article, and cleverly finds a way to bring data to bear on a rather tricky inferential problem. Here are my possibly idiosyncratic thoughts on drawing inferences from the patterns shown and what those inferences would mean.
Suppose that arguments do cause (“affect”) the justices. Then our measure of the treatment is affected by how it affected Justice Blackmun. The scale of the “dose” given is exaggerated. But that, I think, would just lead us to underestimate the treatment effect.
I do see at least one cause for concern. It has to be assumed that the cases getting the different levels of argument quality are assumed to be similar. But it might well be that some cases are easier to vote one way in than the other. The conservative side has the better potential argument. Or the conservative side will have an easier time making the argument fit the particular set of justices on the Court at the time. Then, the quality score given by Blackmun is not a treatment in terms of quality arguments, but rather quality arguments are signals of the underlying case itself. In this interpretation, arguments affect nothing. Some cases simply are easier to decide one way than the other, for the justices in question perhaps, or in general. If so, it still suggests that something legal-ish matters, but it might just be things more akin to case facts than law (whatever those things mean). Or it could be that a lawyer more likely to win will have an easier time at the Court than someone more likely to lose. In the end, I’m not sure convinced that there is a causal finding here and I resist calling the results a test of anything, other than statistical significance of a correlation. But I still do think these results are potentially important in terms of helping us figure out and think about what is going on in judicial decision-making. At the least, they can rule out very naive views of such.
The authors do consider some of my concerns, in footnote 24: “It is possible that attorneys get higher grades in cases in which they have the ‘better’ legal position; thus the relationship we show here could reflect the effect of the legal and factual circumstances of a case. We think that the effect is more plausibly a function of attorney arguments than case facts. First, cases that are placed on the Court’s docket and are decided with an opinion are by their very nature difficult ones that do not result in one litigant clearly having the better side of the case…. [Second…] We are willing to bear the cost of not including facts in our analysis so that we can produce an analysis for the role of oral arguments that is generalizable across issue areas.” I understand the latter point, but the cost of doing that is being able to make a claim for causal inference, in my opinion. About the former point, I agree that generally Supreme Court cases are likely to be difficult cases, but still within this set, it is likely that one outcome is more likely than the other or not perfectly balanced. Within that set, better legal positions might correlate to votes.
Note that it is not clear what information these arguments are providing. It is said in part to be information helpful to bringing policy closer to one’s ideal point, but if you and I disagree to a sufficient extent, it is not the case that this information can be useful to both of us in that way. Suppose the information is clarifying. Again it is not clear why that leads me to vote for that side. One possibility is in line with how I do think about judicial opinions and bargaining, that by increasing the quality of a position, and reducing the error/variance around it, you make that position more attractive all else equal. And so better lawyers will provide more of this variance-reducing legal quality stuff. On the other hand, those with cases that are easier to win can probably better attract high end lawyers.
There are other issues. The way the authors show that Blackmun’s ideology isn’t driving this is to make use of his own ideology score, as is for the conservative attorney and the negative of it when it is the liberal attorney. But this only makes use of the variation in Blackmun’s own score, within the set of liberal-side attorneys and again within the set of conservative-side attorneys. There is no measure of the compatibility of any attorney that varies within each of these sets. (The ideological compatibility score is just measuring the changes in Blackmun’s revealed voting tendencies over time.) Suppose some liberal attorneys are more liberal than others. Then, Blackmun might them a lower score because of greater distance. Then, we think we are controlling for ideology, but are not, and we find that the score matters and read that to mean that quality matters.
All this said, I think it is clear just how tricky all this is to figure out, and I applaud the authors for starting this process. I apologize if I’ve gotten some points wrong in my reading of the article.