My beef with Brooks: the alternative to “good statistics” is not “no statistics,” it’s “bad statistics”

by Andrew Gelman on February 20, 2013 · 18 comments

in Media

Brooks_New-articleInlinestatman

I was thinking more about David Brooks’s anti-data column from yesterday, and I realized what is really bothering me.

Brooks expresses skepticism about numbers, about the limitations of raw data, about the importance of human thinking. Fine, I agree with all of this, to some extent.

But then Brooks turns around uses numbers and unquestioningly and uncritically (OK, not completely uncritically; see P.S. below). In a notorious recent case, Brooks wrote, in the context of college admissions:

You’re going to want to argue with Unz’s article all the way along, especially for its narrow, math-test-driven view of merit. But it’s potentially ground-shifting. Unz’s other big point is that Jews are vastly overrepresented at elite universities and that Jewish achievement has collapsed. In the 1970s, for example, 40 percent of top scorers in the Math Olympiad had Jewish names. Now 2.5 percent do.

But these numbers are incorrect, as I learned from a professor of oncology at the University of Wisconsin – Madison who has published a relevant article in the Notices of the American Mathematical Society on mathematics performance by gender and ethnicity on national and international mathematics competitions. Mertz found, based on her direct interviews with these students, that over 12% (her best guess is something like 16%, I think) of recent Math Olympiad participants were Jewish (and she believes the estimate of 40% for earlier years is too high). It turns out that the numbers Brooks was reported had been constructed from some sloppy counting.

My beef here, though, is not with Ron Unz, who did the sloppy counting. Unz is a political activist and it is natural for him to interpret the data in ways that are favorable to his case. Data analysis can be tricky, and even when people are trying to do their best, it’s easy to make mistakes and to get trapped by one’s own analysis (see, for example, Daryl Bem). It’s hard to get too angry at a political activist for finding what he’s looking for.

And my beef is not with David Brooks for including some faulty numbers in his column. There’s no way he has time to check every claim in everything he reads. There’s no perfect quality control, and the New York Times does not have the research to fact-check every one of their op-ed columns.

No, my beef is with David Brooks for not correcting his numbers. Janet Mertz contacted him and the Times to report that his published numbers were in error, and I also contacted Brooks (both directly and through an intermediary). But no correction has appeared.

The funny thing is, yesterday’s column would’ve been the perfect place for Brooks to make his correction. He could’ve just added a paragraph such as the following:

One trouble with numbers is they can be spuriously confusing. For example, I myself [Brooks] was misled just a couple months ago when reporting a claim by magazine publisher Ron Unz about a so-called “collapse of Jewish achievement.” In my column, I uncritically presented Unz’s claim that the percentage of top scorers in the American high school math Olympiad team had declined to 2.5%. The actual percentage is over 12%, as I have learned from Prof. Janet Mertz of the University of Wisconsin, who has published peer-reviewed articles on the topic of high-end mathematics achievement. The actual data show evidence not of a dramatic “collapse” but rather a gradual decline, explainable by increased competition for a fixed number of slots on the Olympiad team, together with demographic changes.

OK, that’s not so pithy. I’m sure Brooks and his editor could do better. My point is that, if Brooks wants to talk about the limitations of data, he could start with himself.

The problem with Brooks, as with many “quals,” is not that he operates on a purely qualitative level but rather that he does use data, he just doesn’t distinguish between good and bad data. He doesn’t seem to care.

To put it another way, if Brooks wants to claim, of American Jews, that “the fanatical generations of immigrant strivers have been replaced by a more comfortable generation of preprofessionals,” then, hey, go for it. The problem comes in when he supports this claim with bad data.

Just to be clear, I’m not trying to slam Brooks here. I have a beef with Brooks because I think he can do better. I think he’s right that overreliance on statistics can mislead, and I think he could make this point even better by recognizing how this problem has affected his own work.

As the great Bill James once said, the alternative to “good statistics” is not “no statistics,” it’s “bad statistics.”

P.S. I added additional sentences to the inline Brooks quote above in order to provide more context, to clarify that Brooks was presenting the numbers as coming from a particular outside source. It was not right for me to say he was presenting these numbers “unquestioningly,” as he does express some concerns. Brooks expressed some potential criticism of Unz’s conclusions but not of Unz’s numbers. The reason I still think a New York Times correction is in order is that the numbers appear to me to be presented as facts rather than as Unz’s claims. In any case, now that this 2.5% has been refuted, I think it makes sense to correct it. And, as noted above, I think such a correction is in keeping with Brooks’s larger message, which I support, that numbers can be misleading when we don’t know where they are coming from.

{ 18 comments }

Sebastian February 20, 2013 at 2:01 pm

I’m sounding like a broken record here, but I really wish you’d stop using the “qual” label for sloppy punditry (especially because Brooks punditry here was actually of the bad statistics type). Qualitative social scientists are often more attuned to problems with measurement in quantitative data than “quants” see e.g. Munck and Verkuilen on Democracy:
http://www-bcf.usc.edu/~munck/pdf/Munck_Verkuilen%20CPS%202002.pdf
Bowman, Lehoucq and Mahoney on Democracy:
http://www.jamesmahoney.org/articles/Measuring%20Political%20Democracy.pdf
and many more examples.

Andrew Gelman February 20, 2013 at 5:31 pm

Fair enough.

JP February 20, 2013 at 3:16 pm

Great article! I think the problem Brooks has is that he has already come to his conclusion and then searches for the data to support it. It does not matter if the data is good or bad because his mind is already made up. People get this reversees way of thinking all the time from creation scientists describing at fossils to my ex-girlfriend describing my career.

Matt_L February 20, 2013 at 3:42 pm

As soon as you start using words like “some”; “many”; “few” ; “often”; “seldom” or “sometimes” you are using statistics. Nobody is either all qualitative or or quantitative in our reasoning. We all use both. The key thing is to be honest about how we use them to make our arguments.

UptonOrwell February 20, 2013 at 4:56 pm

This may sound a bit cynical, but, having been around many academics, I often get the feeling that “quals” in powerful positions (such as Brooks) experience a sense of insecurity (which is probably unusual for them, given their social standing) when confronted with quantitative research. The learning curve is quite steep for quantitative analysis–both in absolute terms and, especially, relative to qualitative research and/or pure punditry–whether one is attempting to learn it or perhaps even get good enough to be able to critique it methodologically. How tempting it is, then, to dismiss it categorically. This could potentially help explain the aforementioned puzzle of why anti-quantitative people nevertheless quote simple percentages–EVERYONE understands percentages, but not everyone has the will/desire/ability to advance beyond this level of statistics. And, of course, the statistical methods that academics use are only getting more sophisticated, thus further alienating and embittering those who’ve chosen to dismiss it from the outset.

That said, I didn’t get the impression that Brooks was ENTIRELY dismissive, nor does it appear that he is completely ignorant, of some basic stats. But, being in the early stages of studying quantitative methods myself, I can certainly see why some choose to simply dismiss all of it and save themselves a good deal of bewilderment, insecurity, and frustration.

Matt_L February 21, 2013 at 11:54 am

” The learning curve is quite steep for quantitative analysis–both in absolute terms and, especially, relative to qualitative research and/or pure punditry–whether one is attempting to learn it or perhaps even get good enough to be able to critique it methodologically. How tempting it is, then, to dismiss it categorically. ”

Maybe, or perhaps a simpler explanation is that mathematical instruction in the United State is so poor at the primary and secondary levels that the ‘quals’ are left to their own devices. I have a PhD in history, I’ve studied and mastered several languages including German and Hungarian. I’m not stupid, but I’ve struggled with elementary arithmetic since the 4th grade. I should have flunked algebra, geometry and trig in High School, and would have if it wasn’t for extra credit and a large serving of sympathy from the instructors.

Basically, Mathematical education in this country is sink or swim: the students who get it are able to do so because they already have an inclination towards math and they are the instructor’s stars. Those who don’t get the material are ridiculed, demeaned and forced to submit to the same drills that did not help them learn the first time or the twenty third time. After hitting your head against a brick wall in math class for most of primary and secondary ed, it makes sense to stop and go nowhere near the stuff ever again.

So maybe there would be less hostility to and ignorance of quantitative methods if the people who were good at Mathematics and Statistics would actually do a decent job teaching it to the rest of us who aren’t so adept. Personally, I’m really interested in learning more about math in general and statistics in particular, because they might help me better understand the questions I want to answer in my own research. I follow Andrew Gellman’s posts with great interest. But I do not have the time or inclination to go back and repeat Math 100, especially if I have to go through the same hell I did in grade school and high school.

JP February 21, 2013 at 12:58 pm

Matt_L, some friends of mine are the fine folks at Carnegie are very interested in helping people like you learn Statistics (or other stuff). They’ve created online courses that you can access for free. Their aim is to help people who struggle with algebra/math understand statistics.

You can work through the courses as fast or as slow as you like and they include questions that give you immediate feedback and plenty of interesting apps that allow you to manipulate data and see how your statistics change. Here’s a link if you are interested.

http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/

LFC February 21, 2013 at 2:34 pm

UptonOrwell:
This may sound a bit cynical, but, having been around many academics, I often get the feeling that “quals” in powerful positions (such as Brooks) experience a sense of insecurity (which is probably unusual for them, given their social standing) when confronted with quantitative research.

See Sebastian’s comment above. Brooks is not an academic. He is NOT a “qual” or a “quant” or an advocate of “multi-methods research” or a “methodological pluralist” or whatever. He is a newspaper columnist and occasional writer of books who has some intellectual interests (and/or pretensions).

LFC February 21, 2013 at 2:43 pm

So, if the moral isn’t clear, *stop* using “qual” as a synonym for “journalist or pundit or columnist whom I think is sloppy, etc.”

Quite a few journalists have written good works on, e.g., history and/or public policy, and one or two (actually for the modern U.S., I can only think of one offhand, namely Walter Lippmann) have written political philosophy (for lack of a more precise phrase). But nothing is gained by slapping the ugly word “qual” on any of them.

LFC February 21, 2013 at 2:44 pm

Well, perhaps Geo. Will too, but he’s no Lippmann.

Lew Friedland February 21, 2013 at 11:35 am

A more effective route to getting this correction acknowledged might be to contact Margaret Sullivan, Public Editor at the New York Times.
http://topics.nytimes.com/top/opinion/thepubliceditor/index.html

Pat February 21, 2013 at 11:44 am

My husband, a serious quant, likes to say that 95% confidence doesn’t inspire. If one is married to a beautiful woman who is 95% faithful, she will turn down the first 9 men who proposition her, and flip a coin on the tenth.

Emily February 21, 2013 at 12:23 pm

She actually has a (.95^10) chance of being faithful, or almost 60%.

marc sobel February 21, 2013 at 12:52 pm

I think you are giving Brooks the benefit of assuming he is arguing honestly, In fact almost every Brooks column has as a linchpin to his argument a lie or at best a highly controversial opinion, tossed off as if it were a truth universally acknowledged.

Snarki, child of Loki February 21, 2013 at 1:59 pm

Absolutely, Brooks is a font of dishonest argumentation…generally starting with the vapid generalizations, the perspective from 10000 feet, before somehow landing on the precise spot that the GOP is trying to push that week.

But I do think that he (like many other pundits of all stripes) have a large math-phobic insecurity. For good reason, too, because if they wade into a fight with a ‘quant’ (Krugman or Silver come to mind), then they’re armed with a paper-mache water-pistol, while the other side has a tank-mounted flame thrower.

Theophylact February 21, 2013 at 3:49 pm

My impression is that neither the Times nor its public editor deals with errors, deliberate or otherwise, in opinion pieces by its regulars.

brent February 21, 2013 at 5:17 pm

Man, I hate to defend Brooks but I can’t help but notice that he said:

“In the 1970s, for example, 40 percent of top scorers in the Math Olympiad had Jewish names. Now 2.5 percent do.”

and you replied with:

” Mertz found, based on her direct interviews with these students, that over 12% (her best guess is something like 16%, I think) of recent Math Olympiad participants were Jewish (and she believes the estimate of 40% for earlier years is too high). ”

I know nothing about the Math Olympiad, but aren’t “top scorers” and “participants” apples and oranges?

Andrew Gelman February 21, 2013 at 8:28 pm

Brent:

No, the top scorers and the participants are the same in this case. The participants in the Olympiad are those who were top scorers in an earlier test. We’re all talking about the exact same list.

Comments on this entry are closed.

Previous post:

Next post: