Improving Intelligence Forecasting

Sep 7 '12

Mike Horowitz and Philip Tetlock in Foreign Policy:

Academic research suggests that predicting events five years into the future is so difficult that most experts perform only marginally better than dart-throwing chimps. Now imagine trying to predict over spans of 15 to 20 years. Sisyphus arguably had it easier. But that has not deterred the intelligence community from trying; that is its job.

Starting with the 1997 release of Global Trends 2010 — the report that featured the North Korea prediction — the National Intelligence Council (NIC) has repeatedly tried to predict the trajectories of world politics over a 15-to-20-year period. These predictions run the gamut from a 1997 prediction that Saddam Hussein would no longer rule Iraq by 2010 to the more generic prediction of global multipolarity by 2025 in the most recent report. These predictions are the product of hard work by talented analysts who work under political pressures and intellectual constraints. And, in any case, we are skeptical how much better than chance it is possible for anyone to do in forecasting 15 to 20 years into the future.

That said, when we look at these reports in light of recent research on expert judgment, we cannot help wondering whether there are not ways of doing a better job — of assigning more explicit, testable, and accurate probabilities to possible futures. Improving batting averages by even small margins means the difference between runner-ups and World Series winners — and improving the accuracy of probability judgments by small margins could significantly contribute to U.S. national security.

Go read the whole thing.