The Challenge of Measuring Political Ideology

There has been a recent exchange of views about how liberal Barack Obama really is—involving Ezra Klein and Nate Silver, among others.  This gets at the challenge of how we measure political ideology and whether we can make easy comparisons between Obama, previous presidents, other politicians then and now, and so on.  I asked Michael Bailey, political scientist at Georgetown, for his take.  For more on his work on this subject, see here.  Here is Mike:

Ezra Klein recently argued that President Obama is ideologically a 1990s Republican.   Nate Silver wasn’t convinced and argued Obama is simply a 2010s Democrat.  In part Silver relied on Poole and Rosenthal’s DW-NOMINATE scores which show Obama to the left of Johnson and Kennedy and to the right of Carter and Clinton.  Even as he uses these scores, Silver remains skeptical, arguing:

I’m not entirely persuaded that they can capture all of these dynamics. The system is essentially blind to the content of legislation, so if there are changes in the types of bills that Congress votes upon, there could be long-run ideological changes that are not well accounted for by the system. Measuring ideological change is one of the trickiest questions that political scientists face, and a complete treatment would require a thesis- or book-length approach.

Political scientists have long grappled with these issues.  Sometimes it is easy to reify the NOMINATE scores into actual ideology, but even a few examples make it clear this is a terrible idea.
For example, in the 1960s we know politicians moved left.  At the start of the decade, politicians debated fairly mild civil rights reforms in the face of Jim Crow laws; by the end of the decade, politicians debated how aggressively students should be bused to promote desegregation.  But if you look at the NOMINATE scores (in this the “Common Space” version of the scores which are designed to be more comparable across time than the DW-NOMINATE scores) you see nothing of this.  You see something like this figure: racists from the 1960s like Senators Eastland and Ellender are indistinguishable from the modern moderate Democrats like Fritz Hollings.  Eastland openly argued for white superiority on the Senate floor; Hollings voted to override Bush’es veto of the 1991 Civil Rights Act.  There is huge movement but you see none of it in NOMINATE scores:

NOMINATE scores do a reasonable job dividing up members of Congress by ideology within any given Congress, but if we want to answer the kind of question that Klein and Silver are posing, they don’t do the trick.
Hence, the intuition of Silver is correct: if we really want to talk about ideological change, we need to keep track of policies and where politicians move on these policies.  Klein is starting to do that and Silver had some good pushback; hopefully political scientists will be able to put those kind of intuitions into a formal analysis soon.

9 Responses to The Challenge of Measuring Political Ideology

  1. Dale Sheldon-Hess May 4, 2011 at 6:39 pm #

    Didn’t Silver develop a method for comparing baseball players of different eras, by basically chaining along players who played in the same years? (So if player A started in 2005 and is still playing, and player B played from 1999-2004, they can be compared indirectly by comparing both A and B to players who played with both of them, and the process can be repeated all the way back to the beginning of records.)

    Surely Silver possesses the ability to do something similar for NOMINATE… although it would certainly be a time-consuming and difficult task.

    • Jon May 5, 2011 at 4:15 am #

      Dale, the guys who developed NOMINATE, also did this, they are called Common Space scores and can be compared across Congress and chamber. They are a bit more controversial, but you can play around with the data at

  2. geral May 4, 2011 at 7:46 pm #

    I think that there is a typo here:

    “scores which show Obama to the left of Johnson and Kennedy and to the right of Carter and Clinton”

    “left of” and “right of” should be switched.

    • Dale Sheldon-Hess May 4, 2011 at 10:54 pm #

      That’s not it either. The scores actually show Obama to the right of Clinton and Carter (quoted correctly) and to the left of Johnson (also quoted correctly), but to the RIGHT of Kennedy, not the left.

      The original got 3 out of 4 right, you got 1 out of 4 right, so the original was “more right” than your correction.

  3. random nobody May 4, 2011 at 8:02 pm #

    “Surely Silver possesses the ability to do something similar for NOMINATE”

    That’s actually exactly what dw-nominate does. But, like any such process, it can only do so using the data that you actually have, and inevitably with error.

    NOMINATE doesn’t know anything about the content of votes, only who voted which way. Group A always seem to vote together, and in opposition to this other Group B of voters, so groups A and B must lie at opposite ends of some issue dimension. So long as the same people are voting yes together and voting no together, NOMINATE (or any other vote scaling algorithm) won’t be able to see an overall shift to the left or right as John describes.

    • Dale Sheldon-Hess May 4, 2011 at 10:51 pm #

      I was sure that wasn’t the case; I’d only seen NOMINATE scores for a single congress at a time. But sure enough, they do have a “common space” scoring, and it’s even referenced in Silver’s article, although he, for not-well defined reasons, remains skeptical of it (I guess he questions the assumption that views remain stable over time?).

      So I still hold out hope that he will do his own work on the topic to try to correct the short-comings he perceives.

  4. Manoel Galdino May 5, 2011 at 12:32 am #

    At the last MPSA there were a session about quantitative text analys and one paper (by Hans Noel, of Gergetown) used text data to interpret the content of ideal point estimates. The paper was named “Interpreting Legislative Ideal points with help from ideological discourse”, which is the whole point of the debate.


  5. Erik Voeten May 5, 2011 at 2:41 pm #

    Mike is being too modest. Of course he knows how to deal with this problem:

  6. Jon R May 10, 2011 at 4:12 am #

    Erik, doesn’t the problem seem to be less one of technique and more of data? We could use Mike’s work on bridging to figure out where Obama stands relative to other political actors. But our ability to do so — and more to the point — hinges on our ability to use data that most accurately bear on Obama’s ideology. We aren’t lacking for estimation techniques; instead, I think we’re waiting on some enterprising scholar to develop a set of data with which to characterize presidential positions that aren’t plagued by problems of selection and strategy.