Forecasting Revolutions: To (Statistically) Model or Not To Model?

The following is a guest post by Jay Ulfelder.

****

Do recent revolutions in the Arab world show the value or reveal the futility of statistical modeling for forecasting?

In the parts of the blogosphere I watch, it feels like a lot of people have already answered this question in the negative. For example, a February 11 post on Wired.com’s Danger Room blog claims that “America’s military and intelligence agencies have spent more than $125 million” in the last three years on computer models to forecast political unrest, “but if any of these algorithms saw the upheaval in Egypt coming, the spooks and the generals are keeping the predictions very quiet.” On a February 8 post to the The Arabist, Issandr El Amrani brings it more forcefully:

Quantitative analysis and the behaviouralist approach of most American PoliSci academics is a big steaming turd of horseshit when applied in the Middle East. Statistics are useful, yes, when you are in a country that has relevant statistics or where polling is allowed. But things like electoral statistics tell you very little about the political reality of dictatorships, because the data sets are inherently flawed, since they’re either unavailable, fraudulent, or irrelevant.

Frankly, I don’t know how many statistical models have been designed to forecast the kinds of popular unrest and regime change we’ve seen spread across North Africa and the Middle East so far this year. Of the forecasting models I do know about, I don’t know what most of them had forecast for 2011 in the countries where upheavals have occurred. At the same time, I can’t say who qualifies as an expert on politics in those countries, or what those experts have said in the past year or two about the prospects for revolutionary change in the countries they study. Of course, no one else knows the answers to those questions either, so it’s odd to see some people already offering summary judgments about how statistical forecasting have fared in these cases, and how they are faring in comparison to country or area experts. Heck, in most of these cases, we don’t even know the outcomes yet!

Since we can’t systematically compare expert and statistical forecasts of revolutions in the Arab world, I’d like to recast the moment as an opportunity to revisit the pros and cons of statistical modeling for forecasting in a more general way.

The following is a guest post by Jay Ulfelder.

****

Do recent revolutions in the Arab world show the value or reveal the futility of statistical modeling for forecasting?

In the parts of the blogosphere I watch, it feels like a lot of people have already answered this question in the negative. For example, a February 11 post on Wired.com’s Danger Room blog claims that “America’s military and intelligence agencies have spent more than $125 million” in the last three years on computer models to forecast political unrest, “but if any of these algorithms saw the upheaval in Egypt coming, the spooks and the generals are keeping the predictions very quiet.” On a February 8 post to the The Arabist, Issandr El Amrani brings it more forcefully:

Quantitative analysis and the behaviouralist approach of most American PoliSci academics is a big steaming turd of horseshit when applied in the Middle East. Statistics are useful, yes, when you are in a country that has relevant statistics or where polling is allowed. But things like electoral statistics tell you very little about the political reality of dictatorships, because the data sets are inherently flawed, since they’re either unavailable, fraudulent, or irrelevant.

Frankly, I don’t know how many statistical models have been designed to forecast the kinds of popular unrest and regime change we’ve seen spread across North Africa and the Middle East so far this year. Of the forecasting models I do know about, I don’t know what most of them had forecast for 2011 in the countries where upheavals have occurred. At the same time, I can’t say who qualifies as an expert on politics in those countries, or what those experts have said in the past year or two about the prospects for revolutionary change in the countries they study. Of course, no one else knows the answers to those questions either, so it’s odd to see some people already offering summary judgments about how statistical forecasting have fared in these cases, and how they are faring in comparison to country or area experts. Heck, in most of these cases, we don’t even know the outcomes yet!

Since we can’t systematically compare expert and statistical forecasts of revolutions in the Arab world, I’d like to recast the moment as an opportunity to revisit the pros and cons of statistical modeling for forecasting in a more general way.

 

Let me make clear at the start that I’m not a disinterested party. From 2001 until the end of 2010, I worked for a for-profit firm, SAIC, as research director for the Political Instability Task Force, a U.S. Government-funded program that aims to forecast and explain various forms of political change in countries worldwide. I left that job at the start of 2011, but I continue to do similar work as a consultant. In short, I have a vested interest in peoples’ beliefs on this topic. Still, I think that the case for statistics-based forecasting is strong enough to stand on its own. With all of that on the table, here are a few of what I see as the general advantages of using statistical models instead of expert judgment to forecast political change.

  • Clarity. Statistical modeling forces us to organize our information and ideas about what predicts what in a transparent and concise way, and it can quickly give us sharp feedback on how useful those constructs are likely to be in the form of statistical measures of forecast accuracy. To do statistical analysis (well), we have to define clearly and then quantify our measures; scrutinize the resulting data; and choose a model that matches our best ideas about how those data are generated and hang together. Those ideas may turn out to be badly wrong, but statistics can quickly tell us if that’s so, and then we can tinker with the various elements one at a time to discover where we went off track. It is virtually impossible for experts to subject the mental models on which their forecasts are based to the same kind of scrutiny, and my impression is that most experts don’t even try.
  • Comparability. This virtue is really an extension of the first one. Statistical modeling produces forecasts that are directly comparable across cases and over time. Some decision-makers have to consider a lot of cases simultaneously, and they often get their information about those cases from a gaggle of analysts who specialize in specific cases or issues. Other decision-makers may be following a small number of cases or even a single case and want to know whether risks are increasing or declining over time. It’s very difficult to calibrate subjectively generated forecasts from numerous people so they are directly comparable to one another. Analysts may define the same event in different ways, or mean different things by terms like “unlikely,” or exhibit a host of other subjective ticks that make it difficult to compare their assessments. By contrast, that kind of comparability is built into forecasts based on statistical models, which can be used to generate both a point estimate of an event’s likelihood and a mathematical estimate of our uncertainty about that point estimate.

  • Efficiency. Good data does not come cheap, but neither do structured expert judgments. Once we have developed a statistical model for forecasting and have access to updated inputs, the process of generating new forecasts is usually simple and quick. By contrast, the process of eliciting and assembling fresh and comparable judgments from a large number of experts is usually neither.
  • Flexibility. Statistical modeling gives us a structured way to consider alternative futures. Changes in values of predictor variables suggest specific changes in the likelihood of an event’s occurrence. Forecasting models can provide a structure for considering the likely implications of those changes, including events and trends that might not directly affect the risk of the event of interest but could shape the value of those predictor variables.

In the past couple of weeks, I’ve been intrigued to see people who probably would not describe themselves as statistically inclined trying to draw lessons from Tunisia and Egypt by spotting patterns in tables or charts summarizing selected conditions in those countries before their revolutions. These exercises go something like this: “Here’s a list of factors that might have driven those events; there seems to be a pattern in those factors; and, based on that pattern, here’s where unrest is likely to happen next.” Whenever I see that, I think, “Hey, we’ve got a technology that will efficiently find the patterns for you!” Statistical models are usually better at finding those patterns than we are on our own, and they can generate very explicit forecasts when they do. Those forecasts won’t always be correct, but they will find patterns and give formal expression to your ideas in ways that table-scanning or chart-eyeballing rarely will.

I don’t want to sound like the Ginsu-knife pitchman here, though. Even well-designed statistical models have significant limitations, and it’s important to be honest about where those limits lie.

  • Inertia. Statistical models are not good at predicting novel events, or familiar events in novel circumstances. If the conditions under which future events occur don’t resemble the circumstances under which some portion of past events on which the model was trained occurred—in other words, if the future bears little resemblance to the past—then a statistical model is going to be hard pressed to see that future coming. This is the classic problem of induction, and forecasts based on statistical models are vulnerable to this problem by design. Modelers can hedge against this problem by carefully choosing their frames of reference and frequently updating their estimates, but they can never make it go away.
  • Simplicity. Simplicity is often a virtue, but it can also be a vice. One important weakness of statistical models of political change is their inability to reflect and predict the dynamic processes by which that change arises. Models that produce annual forecasts of revolution might prove quite accurate in their predictions and still have next to nothing to say about the causal mechanisms at work and actors involved. Those mechanisms and actors are often of great interest while those changes are occurring, especially to policy-makers contemplating how to respond to unfolding events. By necessity, the crises that statistical models usually forecast well—civil-war onsets, coups, regime changes, and so on—are observable from high altitude, while the day-to-day and even week-to-week or month-to-month events that carry societies to those crises (or not) are things that we still cannot forecast well (and probably with good reason, because the process generating them is, I believe, a lot less deterministic).

I’ll wrap up this guest post by talking about one aspect on which I haven’t tried to compare expert judgments to statistical forecasts, namely, accuracy. Advocates of statistical forecasting often support their position by claiming that, on average, statistical models produce more accurate forecasts than experts do. To be honest, though, it’s not clear to me whether or not that’s true—or, more specifically, under what conditions it’s true. Even if it were true, it’s a generalization, and we can’t know in advance how it will apply to specific problems, like forecasting revolutions (or, for that matter, the outcomes of football games. Just as there is no “average individual,” there is no average forecasting problem, so it’s impossible to talk confidently about relative accuracy without question-specific evidence.

One thing we do know, though, is that the news media pay disproportionate attention to less accurate expert forecasts, and this bias makes it hard for the would-be consumers of these prognostications to figure out to whom to listen. Before naysayers speak too loudly about the flaws they see in statistical forecasting, they might want to think carefully about how to overcome this bias and help decision-makers find useful information in the thick but muddy flow of available expert judgments. Hey, maybe a statistical model could help with that…

7 Responses to Forecasting Revolutions: To (Statistically) Model or Not To Model?

  1. Alex Williams February 15, 2011 at 12:10 am #

    Very good post. I think the criticism of writers for looking for patterns in graphs they put together ad hoc is particularly salient. Two questions:

    1. Is there evidence that expert forecasters are better than statistical models at dealing with the problems associated with inertia and novel events/circumstances?

    2. Is it fair to say that, at least in the majority of the fields where they’ve been compared, statistical models have been more accurate in their prediction than expert forecasters?

    Thanks!

  2. Jay Ulfelder February 15, 2011 at 7:03 am #

    Alex, to your questions:

    1. I don’t know. I have a belief about it, but I don’t know of any empirical evidence on that point.

    2. Again, I don’t know. I’ve seen that claim made, but when I try to source it back to an actual piece of research, I can’t. If anyone can point me/us toward such a source–whatever the finding–I’d be grateful.

  3. Anvar February 15, 2011 at 11:00 am #

    Dear Jay,
    thank you for a well-balanced post. I would like to offer the following comments/questions:
    1. Models might be good for predicting political unrest – i.e. mobilization for regime change, but far worse at predicting the outcome of such mobilization, don’t you think?
    2. We don’t have a dominant theory that accounts for regime change, so I guess quantitative approach is directed more towards opportunistic theory building than actual forecast of events.
    3. Regarding inertia that you have identified as weakness of quantitative approach, experts in my opinion fare not much better. After all, everybody extrapolates the past into the future
    4. Revolutions are a thing of non-democracies, and non-democracies tend to have bad data or no data at all – you still need experts to code some stuff for you – “experts” vs “models” is an ideal-types dichotomy, don’t you agree?
    5. The business of prediction in a complex world, be it complex socio-political events or stock prices, is ultimately impossible not only because humans don’t always act rational but also because of the black swans that usually cause such events
    Many thanks!

    P.S. Will now google the groups you referred to, but can you direct me to any research these guys published?

  4. Alex Williams February 15, 2011 at 11:29 am #

    Jay:

    I looked around a bit and found this meta-analysis.

  5. Philip Schrodt February 15, 2011 at 9:06 pm #

    Alex: Interesting paper — thanks. That 10% figure, curiously, has also been floating around since fairly early in the 2nd wave (1980s) of artificial intelligence, when it was noted that most of the machine learning methods (which, often as not, were being tested on medical diagnostic problems) were performing about 10% better than the human experts. That was just an informal rule of thumb (or perhaps of ten fingers) but this confirms it, at least in that domain.

    [The other interesting result from those early studies was that if the human expert identified some vague “I only know it when I see it” indicator, it usually proved to be irrelevant in the sense that more objective indicators they had also identified provided sufficient classification.]

    On the issue of prediction more generally, the issue that is usually missed is that one wants to get as much prediction as is possible given the nature of the activity, but for rational human political activity, that will never be 100%: there are lots of circumstances that remain for some combination of free will and rational randomness. But that is a long, long way from indicating things are completely unpredictable, and if there are repeated patterns, statistical and computational methods will pick these up. Various medium-term (6 month to 2 year) political forecasting models tend to converge on about 75% to 85% accuracy (again, rule of thumb, and I’m not sure we have enough of those sorts of studies to do a good meta-analysis yet), and so it may turn out that the random component is in the 20% range.

  6. Jay Ulfelder February 16, 2011 at 6:53 am #

    Alex, thank you, that’s very helpful.

    Anvar, on Q1, no, I don’t think statistical models would necessarily be worse at predicting outcomes. In fact, I would expect them to do better for all the usual reasons. If I were designing a study to address that forecasting problem, I would just make the analysis conditional on the occurrence of an uprising. So the cases are countries with uprisings, and the dependent variable is the set of observed outcomes. You could only use initial conditions, or you could do an event history design to allow for time-varying covariates. There are some challenges with analysis of competing risks, but I think there’s no reason to expect models to do especially poorly with this problem.

    I agree with you on Q3.

    On Q4, in my experience, that problem is overstated. Data quality co-varies much more strongly with poverty than regime type. (And that is an important problem, just not the one you suggested.)

    I think Phil covered Q5 nicely.

  7. Anvar February 17, 2011 at 11:57 am #

    Thanks a lot, Jay!