I have a theory

by Hans Noel on November 23, 2011 · 9 comments

in Campaigns and elections

I have a theory about the nerd fight over political science forecasting models.

So far, the debate has touched on a number of very interesting issues, but the thrust is whether political scientists are “too confident” in the models that predict the outcomes of elections on the basis of the economy, or whether there is room for the campaign to “matter.” The conclusion, which most reasonable people seem to agree on, is that of course there is room for the campaign to matter, but the fight continues over what that means about the importance of the economy.

But I think the main source of conflict is a different perspective on why one might run these models in the first place. (And some political scientists may share that perspective.) Political scientists are not in the business of predicting the future.  (I never thought journalists were, either.) Forecasting is good because it keeps us honest. Predicting outside the sample ensures you are not inventing an ad hoc explanation. But models built just to predict are not the point.  Generally, political scientists do statistical modeling because we want to test a hypothesis. And we want to do that because we have a theory.

For instance, we might theorize that voters reward candidates or parties who have served them well, and punish those who have not. One way to measure a high-performing president is by looking at the state of the economy. Another is to look at foreign policy failures, especially wars. Theory says we should think about measures that might be felt by ordinary voters, so disposable income is probably better than GDP, and American casualties are probably a better measure than total casualties. These are the variables in the very influential Hibbs’ Bread and Peace Model.


The model performs well. And from that, we conclude that the theory is likely true. We don’t learn who will win in 2012, because that’s not what the model was for. We learn that the theory is likely true. We then go on to test the theory in many, many, many other ways. We start to believe that the fundamentals of the election, things that have little to do with the campaign and everything to do with what has happened in the last four years, matter a great deal.

This does not mean that nothing else matters, because we have not built a model that attempted to maximize the amount of variation we are explaining. We built a model to test a theory. There are other theories, of course. One is that voters vote primarily on the basis of ideology. This theory has generated mixed results. It predicts, for example, that both parties will nominate centrists. and while presidential candidates are rarely very extreme, candidates from all levels of office are also rarely centrist. So while we think voters do punish candidates who are too extreme, we think parties can get away with nominating someone who is not moderate.

And when we go to test the effect of campaigns, the results are again at best mixed. Campaign advertising definitely moves people, but the effect decays rather quickly, and meanwhile, there are a lot of competing messages. Most of the campaign gaffes that fascinate political journalists and political scientists are often completely unknown to most voters. And the most attentive voters are highly partisan, so they filter those events anyway. In short, while no one thinks the campaign is meaningless, there is little reason to believe it can have the kinds of effects that many attribute to it.

One could build models designed just for predicting the future, and some scholars do. If so, one might include, for example, presidential approval as a forecasting variable. That variable “predicts” very well. But it is theoretically uninteresting. The variables we want to test, like the economic variables, will also affect presidential approval, and so including it hinders our ability to estimate the effect of the variables we care about. Such a model might be great at predicting the future, but it’s lousy for social science.

{ 9 comments }

Ben November 23, 2011 at 2:57 pm

“Political scientists are not in the business of predicting the future…political scientists do statistical modeling because we want to test a hypothesis. And we want to do that because we have a theory.”

Well, okay, but ultimately the point of models, of theories, of science, of knowledge broadly defined, is to guide decisions by make accurate predictions as to the consequences of those actions. It’s great to have a theory that’s true, but it’s not valuable unless it predicts something.

Hans Noel November 23, 2011 at 4:31 pm

Not necessarily.

Yes, a theory needs to have some implications to be meaningful, but it doesn’t need to “predict something” to have implications, and it certainly doesn’t need to predict, with a high degree of accuracy, the dependent variable it was tested on.

In this case, consider the interpretations of election results, which fill news columns after every election. After 2008, many interpreted the result as an embrace of Obama’s policies, and as a repudiation of all things Bush. But since we know that voters tend to punish the incumbent party when the economy is bad, and the economy was oh so very bad, we can explain 2008 as the result of a bad economy, without much appeal to the policy debates of the two campaigns. The voters weren’t necessarily clamoring for universal health care (although many were) or any other policy Obama promised. (Likewise with Reagan in 1980.) The theory improves our understanding, and should guide the decisions of policymakers who are making claims about what the electorate “said” in the election (2008, or in 2010). None of that requires that we can actually predict the outcome of the election.

John Jay November 23, 2011 at 4:32 pm

“It’s great to have a theory that’s true, but it’s not valuable unless it predicts something.”

This was my thought reading this as well. I thought Phil Schrodt argued the importance of prediction to political science nicely:

http://polmeth.wustl.edu/media/Paper/Schrodt7SinsAPSA10.pdf

Excerpt:

My subtle and nely nuanced assessment of [the sentiment that prediction is an inferior to explanation]: This is utterly, totally and completely self-serving bullshit, devoid of even the slightest philosophical justi cation, tremendously damaging to our collective intellectual enterprise, and best dispatched to the trash compactor of history to be revived only as an example of what not to do.

Have I made myself clear?

So, what’s the big deal here?. . . like, just kick back, dude. Okay, okay, bit of a sore spot since I continually get hit by this and yet not once, not once, has anyone been able to provide even a single citation in the philosophy of science literature to justify it. Instead, it is simply folk wisdom derived, I suspect, from the following syllogism

The models we are working with are nearly worthless for prediction
We are valiant scientists of unparalleled intellect and vision
therefore. . .
Scienti c models do not need to predict

Jonathan Ladd November 23, 2011 at 6:20 pm

John Jay,

I think you are misunderstanding Hans and Phil Schrodt’s arguments. It is not that testing out of sample predictions is not useful. But predicting is a means, not an end. Testing predictions is an important way to determine if you have a theory that accurately characterizes the causal processes at play. But developing such a theory is the ultimate end of most positive social science, not prediction per se.

In other words, prediction can be an important tool for evaluating and choosing among theories, but is not an end in and of itself. If you want a citation, try this: http://en.wikipedia.org/wiki/Conjectures_and_Refutations

John Jay November 23, 2011 at 7:29 pm

Jonathan Ladd: I agree with everything you said about predictions as a mean, not an end. I don’t see how that squares with what Hans wrote:

“Yes, a theory needs to have some implications to be meaningful, but it doesn’t need to ‘predict something’ to have implications, and it certainly doesn’t need to predict, with a high degree of accuracy, the dependent variable it was tested on.”

In the example Hans gives, the theory that voters punish the incumbent party for a bad economy is tested every two years. If its predictions started consistently not coming true, we’d have to discard it. So, the argument that the theory “should guide the decisions of policymakers who are making claims about what the electorate ‘said’ in the election” is entirely dependent on the theory’s ability to predict what will happen in elections when the economy is bad.

K. E. D. November 23, 2011 at 10:33 pm

John Jay: You’re making the understandable mistake of assuming that these future events (the ones with which the theory is incompatible) are necessarily commensurable with the theory. The original theory was developed to model some sort of relationship in a certain data set as simply and accurately as possible. This data set bears with it a set of unknown circumstances affecting the relationship in question. However, the breakdown of a theory’s modelling a relationship in an entirely new set of circumstances has no bearing upon the accuracy of the theory in the original relationship, as this new data set bears with it an entirely new set of unknowns. However, the first theory remains perfectly suitable for the data set against which it was first derived.

John Jay November 24, 2011 at 1:45 am

K.E.D.: But by that standard, we end up with a tremendous number of ”theories,” little-to-no ability to preidct future events, and many idiosyncratic explanations. That’s not what I think of when I think of science.

Talleyrand November 26, 2011 at 6:57 pm

So, geology and biology don’t look like science to you? This whole discussion is emblematic of what is wrong with the way political scientists talk about “science” and “philosophy of science”. A theory can be explanatory without also being useful for predicting future events. Future events can be predicted without any understanding of the underlying causal mechanisms. Which is also relevant for this statement of Hans Noel:

“The model performs well. And from that, we conclude that the theory is likely true.”

Not true. This commits the fallacy of affirming the consequent. You can have it explained to you in simple language by Kevin Clarke in The Necessity of Being Comparative: Theory Confirmation in Quantitative Political Science, Comparative Political Studies 2007, 40(7).

John Jay November 28, 2011 at 10:49 am

“So, geology and biology don’t look like science to you?”

On the contrary, if a biologist studies the serotonergic functioning among a group of participants and theorizes about what is happening, it is with the expectation that the theory will predict serotonergic functioning in other people, too.

Comments on this entry are closed.

Previous post:

Next post: