Predicting the Health Care Decision from the Oral Arguments

This is from political scientist Michael Evans, and was originally posted to a law and courts listserv.  I thank him for sending it along:


I was curious about the relative number of words directed at the two sides in yesterday’s oral argument and thought the results would be of interest here.

For those not familiar with the research on this (see below), it has shown that Justices tend to direct more questions and words at the side they eventually vote against. (Questions and words are highly correlated, but I prefer words because questions are harder to define.) The theory is that Justices generally do not play “devil’s advocate”—asking questions to help the side they support—but, rather, attempt to expose what they see as the weaknesses of the other side’s arguments.

This table shows the relative number of words uttered by each Justice to the two sides regarding the constitutionality of the individual mandate under the commerce clause. As is typical, Thomas did not ask any questions, but I think his vote is “somewhat” predictable. I excluded questions about the taxing power because, tellingly, no one asked Clement or Cavin a single question about the taxing power:

For what it’s worth, here is my take. Based on this, I think (even more than I did after reading and listening to the oral arguments) that the individual mandate is in serious trouble. This indicator suggests a 5-4 decision striking down the mandate. Of course, it goes without saying that the number of questions/words is not a perfect predictor (~86% accuracy in the published studies). And the most important caveat here is that Kennedy is the least predictable Justice (according to Epstein et al. 2009) and the discrepancy in his questions was lower than the other justices’. Another important caveat is that the aggregate results actually suggest a slightly higher probability that the mandate will be upheld (54% vs. 46% of words).

However, when we look at individual justices, which seems more appropriate here, the Court appears sharply polarized along familiar ideological lines and Kennedy uttered just over twice as many words to Verrilli as he did to Clement/Carvin, which is a greater discrepancy than is typical for him (according to Epstein et al.). For all other justices, moreover, the discrepancy was much starker than the average discrepancy reported by Epstein et al.; and the latter also found that the number of questions and words uttered by conservatives is a fairly good predictor of Kennedy’s (and Thomas’) vote. And note, finally, the whopping 39 percentage point difference between Kennedy and the closest liberal justice (Ginsburg) vs. the 13 percentage point difference between Kennedy and the closest conservative (Alito).

Again, this is just my take. I’d be interested to hear reasons to doubt this interpretation of the data, especially by those who have studied the predictability of votes based on oral arguments

Here is the list of studies:

  • Epstein, Lee, Landes, William M. and Posner, Richard A. 2009. “Inferring the Winning Party in the Supreme Court from the Pattern of Questioning at Oral Argument.” University of Chicago Law & Economics, Olin Working Paper No. 466. Available here or here.
  • Johnson, Timothy R., Black, Ryan C., Goldman, Jerry and Treul, Sarah. 2009 “Inquiring Minds Want to Know: Do Justices Tip Their Hands with Questions at Oral Argument in the U.S. Supreme Court?”9). Washington University Journal of Law & Policy, Vol. 29, 2009.  Available here.
  • Shullman, Sarah Levien. 2004. “The Illusion Of Devil’s Advocacy: How The Justices Of The Supreme Court Foreshadow Their Decisions During Oral Argument.” Journal of Appellate Practice and Process. Vol. 6: 2.  Available here.
  • See also Lindgren, Jim. 2005. “Does Asking More Questions Tip the Outcome of Supreme Court Cases?” Volokh Conspiracy.

13 Responses to Predicting the Health Care Decision from the Oral Arguments

  1. LFC March 29, 2012 at 12:03 pm #

    Esp. in a case like this, wouldn’t it make more sense to consider the substance of what the Justices asked/said, not just count how many words they directed at each side, even if word counts have been shown to have some predictive value?

    I listened to parts of the argument (on C-span radio) and my impression matched what Walter Dellinger subsequently said on the NewsHour: it was pretty clear from the substance of the questioning that the four more ‘liberal’ justices were inclined to uphold the individual mandate, whereas Scalia, Alito and (presumably) Thomas are going to vote to strike it down. Kennedy and Roberts, however, were directing fairly tough questions to both sides on that issue, making it, based on the substance of the questions, harder to predict how they are going to go. Roberts, for ex., seemed concerned that the mandate required people to buy particular types of insurance coverage they weren’t going to need, e.g., an adult with no children would have to buy a package including pediatric coverage (according to Roberts’s questions). Whether you can derive a prediction about his vote based on this line of questioning I don’t know, but it’s certainly not the kind of analysis that a word-count is going to help with.

    I don’t study the Sup Ct and am not familiar with the cited articles, but it seems to me that counting the number of words directed at each side is a somewhat reductive approach that might raise some problems. However, I’ll let others who are more knowledgeable debate that.

  2. Andreas Moser March 29, 2012 at 12:23 pm #

    I am not sure this study helps. We knew before that and how the Supreme court is divided. No surprise there.

    I’ll just wait until the decision comes out.

  3. RobC March 29, 2012 at 3:08 pm #

    Although Michael Evans refers here to “yesterday’s oral argument,” may we assume this was written on Wednesday and refers to Tuesday’s oral argument? It was on Tuesday, of course, that the constitutionality of the mandate was argued. Wednesday’s arguments were about Medicaid expansion and severability.

  4. Robert Hume March 29, 2012 at 11:06 pm #

    The frustrating thing is that Kennedy did not give us much to go on. As these numbers suggest, besides Thomas, Kennedy participated the least in the argument. And while he directed some tough questions at Verrilli, Kennedy was ambiguous enough in some of his other questions to keep us guessing.

  5. Jackson March 29, 2012 at 11:16 pm #

    Does anyone else find it odd that we suppose that the Court behaves in a certain manner while at the same time completely disregarding the actions of Thomas? We predict that the Court will behave in a certain manner based upon questions, or in this case number of words, but at the same time we allow that the decision of at least one of The Nine is predetermined based upon political leanings? If it’s perfectly acceptable (as it seems to be a foregone conclusion as to how he will vote) for Clarence Thomas to rule the way he does on whatever issue is before him, how do we pretend that the rest of the Court behaves differently?

    • Scott Monje March 30, 2012 at 2:19 pm #

      Thomas’s voting has been pretty consistent. On the other hand, a former Kennedy clerk on NPR the other day said that he can never predict how Kennedy will vote.

  6. thomas March 30, 2012 at 1:30 pm #

    One thing to remember of the supremes is the red hinie given to them a the state of the union adess by Obungle, this has been rolling around the craw growing more and more bitter with time. One can also keep in mind the dressing down the obungle adminstration gives the court at every turn. I would think a landmark year and a half kissing of the hind end of the special intrests that crafted this law without imput from lawmakers (remember no one read this monster before voting on it) will be a perfict opertunity to show the axium (never f_ck with the ships electician) to obungle while laughing at him infront of the whole world. ONE for the supreme court. OK for the real jest of what is going on. The democrat controlled congress (all branches of the government had a democrat margin large enough to push anything thru without a single vote from the republicans) pushed this through , make it law without reading any of the whole 27000 page boondogle. They passed a un-constitutional law with thier eyes wide open, ignoring the foundation of law the country was based on. NO ONE READ THIS BARSINSTER OF A BILL/LAW ,so the Supreme court, an equal branch of the government will call this black death of a law what it is ; UNCONSTITUTIONAL as it should be. Thank goodness we still have some of the constituion left to use after this and previous adminstrations have stomped on it over and over again to the point of advising other emerging countrys not to use it as a reference for crafting thier own. This comming from a democrat liberal apointee and now justice who swore on the BIBLE to uphold and protect the constitution , she should be brought up on charges of sadition and treason and vioating the terms of her sworn testomony at her sware in.

    • LFC March 31, 2012 at 6:44 pm #

      This marginally literate comment by “thomas” is a piece of complete garbage

  7. tk March 30, 2012 at 2:42 pm #

    Regardless of the vote, it cannot work. There is no money to pay for it and health care will get far worst with the end result of implosion of the federal government. If liberals had any intelligence at all, they would want it repealed for it has a more disastrous effect on them and their voting constituency. The rich will just get their health care outside the country. And there will not be enough TSA agents to stop them assuming they can get paid. People mistakenly say that this will change the relationship forever between citizen and government. This is wrong as the actual death came about due to the civil war. This is when everything change. However, the noticeable death of the nation came with the introduction of fiat money, the federal reserve, and income taxes. But hey, who really cares about history, right? Especially when it shakes so many misconceptions. Who care if the last thirty years of economic history was bought about because of an economic and the the illusion of it by lose loan standard, debt, inflation, and money created out of thin air.

  8. Hersh Adlerstein March 30, 2012 at 3:41 pm #

    Funny, I thought there were 9 Justices, or is Thomas really the Invisible Man. Does he march in such lock-step with Scalia that he has no ideas of his own?

    • RobC March 30, 2012 at 4:34 pm #

      I don’t think it’s fair to elide Thomas’s choice not to ask questions at oral argument with having no ideas of his own. Thomas’s opinions suggest otherwise, whether you happen to agree with his ideas or not. To be sure, it’s entertaining to pepper with questions counsel whose arguments you disagree with, and it may serve some value in exposing the weaknesses of the argument to those who might otherwise be inclined to accept it, but engaging in trial by verbal combat is by no means obligatory.

      • LFC March 31, 2012 at 6:48 pm #

        RobC is right. Thomas does not always agree with Scalia. (His views are sometimes more, I would say extreme, a word that reflects my own views, to be sure.)

  9. Brian Schmidt March 31, 2012 at 11:32 am #

    I blogged about this here:

    The key issue is Kennedy, but according to Epstein, Kennedy’s questions/words aren’t predictive. So the upshot is that we still don’t know.

    What I don’t get is that given Kennedy is so decisive to the Court, I’d expect him not being predictive would make the Court itself not very predictive (except as Epstein notes in the unanimous opinions).