The ABA Prefers White Men

Maya Sen (University of Rochester) shows in a working paper (pdf, ungated) that the American Bar Association (ABA) has systematically assigned lower judicial qualification ratings to minority and female nominees for U.S. District Courts. This finding holds after controlling for observable characteristics (via matching) that may be associated with qualification; such as the quality of the law school where a nominee got her law degree, how much experience she has practicing law, and whether this experience was attained as a law clerk, a public defender, a judge (of several varieties), or in some other role. Partisanship plays little role in the qualification ratings for these lower Court judges (in contrast to appointments to Federal Courts of Appeals).

Sen also shows that getting a lower qualification rating is correlated with a reduced confirmation probability. On the other hand, qualification ratings are not correlated with an indicator for judicial quality (high reversal rates). This suggests that the ABA does not have some sort of private information about the true quality of judges that is not captured by observable characteristics of judges.

These findings are important not just because of the direct effect of qualification ratings on the composition of district courts but also for what they say about the process of how the legal profession evaluates qualification more generally. The ABA rates judicial candidates in a non-transparent way on subjective qualities such as their “judicial temperament.” But what is temperament? Is it possible that temperament becomes associated with certain mannerisms of respected judges who all happened to be white men? And why does the ABA gets to play this role anyway? Is there any evidence that the ABA has private information that once revealed would make the functioning of the judiciary better? This question is important too because there is ongoing discussion about the benefits of having judges appointed via professional associations rather than politicians or electorates.

One downside of the paper is that it estimates an average effect between 1960 and 2012. It is plausible that things have changed in this time as more women and minorities have set examples as judges and have entered the ABA’s evaluation committees. This is difficult to study given the somewhat low numbers of female and minority candidates. Still, I was struck by how the Standing Committee on the Federal Judiciary now defines “judicial temperament:”

In evaluating “judicial temperament,” the Committee considers the prospective nominee’s compassion, decisiveness, open-mindedness, courtesy, patience, freedom from bias, and commitment to equal justice under the law.

I doubt this is how they defined it back in 1960 (of course there is no guarantee the actual Committee pays any attention to this definition). In any way, I really enjoyed this paper and would like to see more systematic studies of the role of professional associations.

ps. some small corrections added (see comments)

10 Responses to The ABA Prefers White Men

  1. Leslie February 28, 2013 at 9:35 am #

    Presumably the parenthetical comment ending the first paragraph was intended to refer to Federal Courts of Appeals.

    • Erik Voeten February 28, 2013 at 10:24 am #

      Thanks, yes, corrected.

  2. RobC February 28, 2013 at 9:42 am #

    Something seems to be missing or confused in the last sentence of your first paragraph. As I’m sure you know, district court judges and appeals court judges are all Article III federal court judges. Also, if what you mean to say is that partisanship plays little role in district court appointments, I’m a little surprised at that statement and wonder if there are data supporting that.

    • Erik Voeten February 28, 2013 at 10:27 am #

      I should have said that partisanship plays little role in the qualification ratings that the ABA gives to district court judges (unlike for Court of Appeals judges). Clearly should have had more coffee this morning. I corrected the sentence.

  3. Andrew Gelman February 28, 2013 at 11:26 am #

    “One downside of the paper is that it estimates an average effect between 1960 and 2012.” indeed! Since (I assume) the paper has not yet been published, the natural way to address that would be to include time in the model. This should be easy enough to do. Before doing that, a short-term fix to the paper would be to replace the present-tense verbs to various past tenses, for example changing “are” to “have been” in the sentence “minority and female nominees are more likely than whites and males to receive these lower ratings.”

    • Andrew Gelman February 28, 2013 at 11:27 am #

      P.S. Along the same lines, I’d recommend changing “prefers” to “has preferred” in the title of the post, considering that 0% of the members of the ABA committee today were performing these evaluations in 1960!

  4. John Griffin February 28, 2013 at 12:41 pm #

    I think there are two different interpretations of the core finding here.

    What I take to be the author’s interpretation is that the ABA ratings are influenced by implicit bias or discrimination such that apparently equivalent candidates are rated differently based on ascriptive characteristics. This is certainly possible and if true would be troubling.

    Another possibility that can’t be ruled out is that the ABA ratings weight the education and experience of some candidates differently than the same education and experience acquired by other candidates because of preferential admissions or hiring schemes. A matching approach would not be able to rule out this possibility.

  5. Rachelle February 28, 2013 at 1:10 pm #

    One cannot help but wonder if the unrelenting pressure for affirmative action has produced a crop of women and minorities who are, in fact, less qualified on average than men.

    You cannot ‘celebrate diversity’ and then expect identical results.

  6. Lawrence Zigerell February 28, 2013 at 2:08 pm #

    Evaluating whether the ABA issues biased ratings requires modeling the characteristics that should matter and then adding characteristics that should not matter, such as race, gender, and partisanship. The ABA-stated criteria are integrity, professional competence, and judicial temperament. But the model includes variables that touch on only one element of only one criterion.

    Consider the ABA discussion of its rating of this “not qualified” candidate: The report listed allegations that — among other negative things — the candidate in service as a judge was rude, unreasonable in scheduling, had been “spitting angry” on occasion, and wrote well generally but wrote confusingly when the issues were complex. None of these complaints is reflected in the model, so it is perhaps incorrect to consider this candidate to be similarly-situated to a candidate who received no complaints.

    The core problem is that easily-measured quantitative variables are being used to assess a highly qualitative judgment. The sheer number of observations might prohibit a full qualitative analysis but does not prohibit *any* qualitative analysis. Perhaps a section could be added that provides a qualitative follow-up in which a small number of randomly-selected female and African-American candidates are compared on measures of perceived integrity, perceived professional competence, and perceived judicial temperament to a few randomly-selected white male candidates with similar professional backgrounds: the decrease in reliability would be at least partially offset by an increase in validity. Or — if a substantial number of ABA statements are available — perhaps the quantitative model could incorporate measures of perceived integrity, perceived professional competence, and perceived judicial temperament.

    This type of analysis would be more convincing and more charitable to the ABA. In the case of the candidate mentioned above, an ABA representative reportedly interviewed 65 persons who had interacted with the candidate, presumably to gather information that could not be found through automated content analysis of data available at the Federal Judicial Center. Perhaps it is unfair to complain that the ABA process is not transparent enough (p. 37) if relevant information that the ABA *has* made public is not incorporated into the analysis.

  7. Lawrence Zigerell February 28, 2013 at 2:14 pm #

    The quantitative model could also be tweaked to be more representative of how the ABA considers or should consider candidates: for instance, it does not appear correct that the 25th ranked law school should receive a weight different than the 26th ranked law school but the same as the 1st or 2nd ranked law school, it does not appear correct that Supreme Court clerkships receive the same weight as lower federal court or state court clerkships; and it does not appear correct that state high court judges receive the same weight as state lower court judges.