Education could use some systematic evaluation

David Brooks writes:

There’s an atmosphere of grand fragility hanging over America’s colleges. The grandeur comes from the surging application rates, the international renown, the fancy new dining and athletic facilities. The fragility comes from the fact that colleges are charging more money, but it’s not clear how much actual benefit they are providing. . . .

This is an unstable situation. At some point, parents are going to decide that $160,000 is too high a price if all you get is an empty credential and a fancy car-window sticker.

One part of the solution is found in three little words: value-added assessments. Colleges have to test more to find out how they’re doing.

I agree with that last paragraph. Eric Loken and I said as much in the context of statistics teaching, but the principle of measuring outcomes makes sense more generally. (Issues of measurement and evaluation are particularly salient to statisticians, given that we strongly recommend formal quantitative evaluation in fields other than our own.)

I don’t have anything to add on the substance (beyond again expressing my agreement on the desirability of empirical measures of student performance) but I do want to hypothesize on the sources of Brooks’s doomy impressions.

After all, on first impression, top public and private colleges and universities are doing well, and there’s a lot of demand for their services. Just for example, when I was growing up in suburban Maryland, it was my impression that lots of students who were in the middle of the pack academically in high school could graduate and go to the University of Maryland in nearby College Park. In the decades since, the University of Maryland has become more competitive, reflecting the increasing demand (not matched by increasing supply) for high-quality college education.

So why is Brooks so sure that universities are in trouble? Why paint their current success as an “it’s always brightest just before the dark” situation (to borrow the words of Jim Thompson) rather than a more conventional presentation of universities as a shining success?

I see three reasons. I don’t know that any of these are conscious on Brooks’s part, but I suspect they all went into his reasoning.

1. Politics: Brooks is a political conservative. Universities are bastions of liberalism, thus it is pleasant of Brooks to see universities as struggling institutions in need of radical change.

2. Personal experience: Brooks has worked all his adult life at newspapers. Newspapers twenty years ago were where universities are now. Newspapers were making tons of money (from the business press, I recall that “Wall Street” was demanding 15 percent annual returns, and newspapers were delivering), in many cases they were close to local monopolies (consider the Washington Post) and successful both financially and in the sense of doing their job well (delivering lots of news), but there was a sense that this was all going to end. Not too many people were starting new newspapers, which was a bad sign, and many people were (correctly) worried that the social and economic basis for newspapers was disappearing. We no longer need to spend that quarter a day to read Peanuts, Art Buchwald, and the story of the latest Redskins game, and advertisers can reach us from other directions.

In short, Brooks has seen it happen in slow motion right in front of him, so he’s primed to see the same pattern of gradual and then panicked decline elsewhere.

3. Economics: Universities are indeed doing well intellectually and financially, but much of that comes from government support. Consider three leading sectors of the economy in the past twenty years: education, health care, and government (including the military). What do they all have in common? Guaranteed or near-guaranteed flows of tax money.

Just for example, one small contribution to the prominence of Columbia University is my research and this blog. I have time to do all this (rather than, for example, spending 40 hours a week grading papers) partly because I have millions of dollars of government grants. (And the granting agencies give extra funding to the university, so my grants also helps support the work of my Columbia colleagues who are not externally funded.) I think this is a good use of tax dollars—-but of course I’d say that, just as Gen. Ripper supports the use of taxation to pay for expensive bombers. My point here is not to argue the merits of the case, just to point out that much of the financial success of universities relies on public funding for research, student loan guarantees, etc.

18 Responses to Education could use some systematic evaluation

  1. John April 20, 2012 at 9:58 am #

    This kind of speculation on personal motives feels beneath this blog. You don’t have to dig far into Brooks’ work to find he frets about higher education because, to him, it resembles a bubble economy. That seems to be his likely motivation.

    • Andrew Gelman April 20, 2012 at 2:39 pm #


      I didn’t say anything about Brooks’s personal motives. I was talking about his personal attitudes and his personal experiences, and that seems fair enough to me. Also, see item 3 above, which I think covers some of your comment about the economics.

  2. LFC April 20, 2012 at 10:55 am #

    I have time to do all this (rather than, for example, spending 40 hours a week grading papers) partly because I have millions of dollars of government grants.

    So your grants pay, among other things, for teaching assistants to grade the papers? You’re allowed to use the grant money to hire more TAs than the university would otherwise provide? (I don’t know much about this, just asking.)

    • Andrew Gelman April 20, 2012 at 11:03 am #


      The grants don’t directly pay for teaching assistants. They pay the university enough so that we only have to teach 3-4 classes per year. If you kept Columbia as it was but suddenly zeroed out all the government grants, the university would not be able to afford paying us good salaries to do our particular combination of teaching and research.

    • Sebastian April 20, 2012 at 6:00 pm #

      To clarify what Andrew is saying – Universities keep a large chunk of grants that go to their faculty – I believe something like 20% isn’t uncommon.

  3. del2124 April 20, 2012 at 11:01 am #

    But the problem is that college is too expensive, not that we somehow don’t know what it gives students (at its most basic level it provides students with the credential they need to get a job). Value-added assessments won’t make college any cheaper.

    • Andrew Gelman April 20, 2012 at 11:07 am #


      I think Brooks is making the argument that good evaluations would lead to healthy competition among universities. Sure, if enough students are willing to pay $40K for “Harvard” or “Swarthmore” no matter what, then it wouldn’t matter how the education there is going. But Brooks is implicitly making the reasonable claim that, with the (hypothetical) availability of good outcome measures, universities will be motivated by economic pressure to improve their educational efforts.

  4. AGH April 20, 2012 at 12:29 pm #

    Dr. Gelman, do you have any thoughts about the recent NYT roundtable about tax breaks for wealthy colleges?

    • Andrew Gelman April 20, 2012 at 12:48 pm #


      Personally, I think colleges and universities serve a valuable function and I am supportive of policies that support them. I don’t mind that some of the subsidies go to wealthy institutions. Similarly, I like that the city keeps the streets paved, the garbage collected, and the trains running, and I don’t mind that various people use these services. I do not oppose a policy just because rich people and rich institutions benefit from it. To me, the decision should be based on the merits of the case; I would consider redistribution as a separate issue.

  5. Matt_L April 20, 2012 at 2:00 pm #

    I appreciated the candor of your column in Chance. Indeed, it is very easy to prescribe a does of quantitative assessment, when the ox that gets gored is not your own.

    That said, as someone who teaches in the humanities, I have often found this question of value added assessment very frustrating. Its hard to reduce the skills and knowledge we teach in a history classroom to strictly quantitative measures that are readily understood by people outside the discipline. And yet it has proven very difficult to convince my colleagues to agree on specific outcomes that can be assessed qualitatively.

    In part this is the result of muddle headed thinking about what history, or the humanities in general, brings to the party. But the reluctance also stems from the understanding that anything we produce for assessment can and will be used against us by the administration later on down the road. As important as it is to improve teaching and curriculum, we are pretty sure that if we admit any shortcomings, we’ll be punished in the name of ‘accountability.’ Given the punishments dolled out to primary and secondary education institutions under NCLB, this fear does not seem unwarranted.

  6. Rhonda April 20, 2012 at 2:29 pm #

    Prof. Gelman: I’m confused. Brooks is a conservative, yet wants radical changes? I thought conservatives dont like radical changes to society. Please clarify this statement, Prof. Gelman. Thanks.

    • Andrew Gelman April 20, 2012 at 2:42 pm #


      Conservatives often support changes that lead to reduction of liberal inference. For example, conservatives tend to support restrictions on labor unions. Conservatism does not mean keeping everything the way it currently is. (Similarly, liberalism does not imply automatic desire for changes in all institutions.)

      • Tad F. April 22, 2012 at 6:28 pm #

        Once upon a time, in political science, “liberal” and “conservative” were points on the rate of change spectrum. “Radical” and “reactionary” being the respective end points on that line.

        It’s true that the terms have other meanings, but Rhonda is correct that in so far as we are taking the classic poli sci meaning.

  7. Jim April 20, 2012 at 2:38 pm #

    Value-added assessment is a terrible measure, and would make even less sense in a university setting.

  8. Andrew Perrin April 21, 2012 at 8:50 pm #

    I tend to think Brooks says stuff like this because Brooks enjoys writing things that seem clever but are, at baseline, wrong. College educations are too expensive, but largely because a significant part ought to be understood as a public good and therefore publicly supported, and this public support is disappearing fast. (This is true of privates too, but of course more so of public ivies like mine.)

    I’m not fundamentally against systematic evaluation, but very, very skeptical that it can be accomplished for what are among the most important courses in college: humanities, interpretive social sciences, and syncretic interdisciplinary courses. I could be convinced that it’s possible, but if it’s not, I think it’s far better to teach well and evaluate poorly than to teach poorly and evaluate well.

    • Andrew Gelman April 21, 2012 at 9:51 pm #

      Andrew P.:

      You might be right on that. I’m pretty sure the classes I teach could be improved via systematic evaluation, but maybe I’m leaning too heavily on my own experiences. Also based on my experience, I don’t see teaching and evaluation as competing with each other. On the contrary, I think better evaluation would motivate me and others to teach better. Even without the threat of losing our jobs; the evaluation would be enough!

  9. Noumenon April 23, 2012 at 9:33 am #

    Do you personally have millions of dollars in grants, or your whole university? I didn’t really know that statisticians got government research grants at all. Forgive me for not understanding what you do. Do the grants buy computers, or pay people to prove theorems, or what?

  10. Keith November 4, 2012 at 10:25 pm #

    Higher education institutions are becoming increasingly important for training America’s workers for the jobs of tomorrow; empirical tests should be applied to maximize the value students are receiving.