Convergence between Polls and Prediction Markets in US Presidential Election

by Joshua Tucker on October 18, 2012 · 22 comments

in Campaigns and elections,Methodology

I’m intrigued by the recent convergence between Nate Silver’s 2012 US Presidential Election forecast (which incorporates most of the publicly available polling data) and the futures contracts being sold on an Obama victory at Intrade. Here’s Nate’s most up to the minute forecast (Obama in blue):

And here’s the current cost of a contract that pays $10 if Obama wins at Intrade:

They are, by all accounts, stunningly close to each other. I imagine that there are four possible states of the world that could be generating these results using two completely different methodological procedures.

First, the true state of the world could be that Obama currently has a 65% chance of winning the election, and both Nate’s approach of aggregating across polls and taking account of economic conditions and vagaries of the campaign AND the ability of traders to correctly price risk in the presidential election could be very effective tools in estimating this true probability.

Second, it is possible that both the polls and the prediction markets are suffering from the same sort of biases and arriving at the same conclusion. For example, the true state of the world might be that Obama has an 80% chance of winning the election, but both Nate’s method and the traders are underestimating the participation in the election of people who no longer have a land-line phone.

Third, it may be the possibility that the same factors (e.g., poll results) are driving both Nate’s results and the decision of the traders regardless of the actual state of the world. Maybe traders are watching the exact same polls as Nate, and therefore coming to the same conclusions.

However, I’d like to suggest a fourth possibility (gleaned from reading the comments on the intrade market), which is that maybe a lot of traders are actually reading Nate’s blog. On the one hand, it seems unlikely that enough people participating in a market are getting their information from the same source. And we know that in markets that a lot of factors come into play besides objective information. However, at the same time people who are betting in these markets are risking their own money, which suggets an incentive to seek out information about the state of the race. And these days, an awful lots of roads on the internet lead back to Nate’s blog.

Why should we care if this is the case? I think there are two potential important implications if this fourth possibility is in play. First, it suggests that prediction markets may be less of an “alternative” guess about the outcome of the presidential elections as compared to polls as they may have been in the pre-538.com days. Second, it might be important to account for this kind of feedback loop in the days after the election when we are trying to assess the prowess of various forecasting methods. In a sense, it reminds me a little bit of when I read papers that cite various expert surveys regarding the policy positions of political parties in different countries as having “additional evidence” because they use multiple expert surveys without realizing that those “different” expert surveys are probably talking to a lot of the same people. If Nate’s blog has achieved the iconic status among election watchers that I suspect it may have, then comparing his predictions to the predictions of futures markest like Intrade may not be bringing as much new information to the table as we think. From a pragmatic standpoint, Nate might want to pause before citing congruence between his model’s predictions and pricing in futures markets as a validation of the model. Ironically, this might be a sign not of the model’s validity, but of the blog’s influence!

The challenge to this interpretation, of course, is that the markets and Nate’s blog have not always converged over the course of the campaign. So a next step might be to think theoretically about the sorts of factors that make traders more likely to be attention to the blog; interpreting the effect of big events like debates seems a useful place to start.

{ 22 comments }

Nate October 18, 2012 at 8:05 am

Well, one other possibility is selection bias. Convergence between the two is pretty salient: it’s perceived as more notable than the times where they differed through the campaign (typical difference of maybe 3-15 points?). But this seems like a pretty empirical question to me. If 538 is getting more influence in the betting markets when people in general pay more attention to the election, I would expect the divergence to start high and get smaller as the election approaches (I think proximity to the election would be a better predictor than single events in the influence of 538). If instead this is a case of selection bias, there should be no solid trend in the convergence over time – it should come and go. One other thing to look at would be the convention “bounces” and the october 2 debate. Nate’s models have a lot of protection against noise, and there’s no way for him to “tell” his model that some sudden movement in the polls is happening for a reason (rather than just randomness in the tracking polls). That means it can take Nate’s model a few days to catch up to the “real” state of the race after a sudden shift. If intrade is particularly influenced by 538, they too should have a slower reaction than the actual tracking polls. How much does their movement precede Nate’s?

Matt October 18, 2012 at 9:29 am

If Nate will play along, there’s a simple test for this — just publish some strong fake results on 538, and see how the Intrade market responds.

It has been pointed out in numerous places that Intrade is very illiquid, and that the British gaming houses are much better positioned as prediction markets, given both the volume of trades and the existence of very large bets.

Andrew Gelman October 18, 2012 at 11:02 am

“65.7%” seems a bit hyper-precise to me. What could it possibly mean to compute this probability to that level of precision?

Jeffrey Friedman October 18, 2012 at 11:31 am

What could it possibly mean, period, given that this election will never be repeated?

Andrew Gelman October 18, 2012 at 12:56 pm

Jeffrey:

Probabilities can have a lot of meaning, even for an event that will never be repeated. For example, suppose you want to make a decision where the outcome is contingent on who wins the election. It can make sense to quantify your uncertainty using probability. Neumann and Morgenstern wrote a book about this! But at some point the quantification becomes meaningless. “60%,” sure. “65%,” maybe. “65.7%,” no way.

benjamin November 8, 2012 at 8:16 pm

“80%” in “the true state of the world might be that Obama has an 80% chance of winning the election” does not seem to be used for quantifying uncertainty, though?

Byron Delaney October 18, 2012 at 11:40 am

What if you had a 35% chance of winning the lottery? Also, economic conditions are way worse than the federal government lets on. And there’s all sorts of nuances that people will respond to that change their minds that can’t be accounted for.

Sebastien Turban October 18, 2012 at 1:42 pm

One thing one can do is check the timestamps on Nate Silver’s posts and look at the time and sales on InTrade. I checked that a couple of days ago (I don’t know how to get trade by trade data for more than 24 hours on the Obama winning contract), Nate posted at 6.42am on October 15th and at the same time, there was a significant fall in the Intrade price. I did that because something similar happened a couple of days before on October 13th when Nate posted at 7.34pm. (except that the price rose — the first line of the post was “President Obama halted an 8-day winning streak for Mitt Romney “)

RobC October 18, 2012 at 3:01 pm

I’ll confess to being a little surprised that you consider the possibility that Silver’s predictions have a significant influence on Intrade trading as the fourth possible explanation, and one that required gleaning the comments of Intrade participants. It took you that much detective work and head-scratching to arrive at that possibility? It’s not as if his blog is a well-kept secret. It’s published on the website of the New York Times, for chrissake.

Max Timmons October 19, 2012 at 12:46 am

Intrade and 538 diverged considerably today though casting some doubt on this hypothesis. 538 is current at 70.4% chance of Obama winning and Intrade is at 62.3% chance of Obama winning.

Jeffrey Friedman October 19, 2012 at 9:16 am

I know there’s a vast literature on this, but I’m still curious, as a non-statistician, what it could mean for there to be a meaningful 65% probability (as opposed to a non-quantifiable likelihood) that a one-time outcome will occur. If 65.7 is too precise, why isn’t 65?

My hypothesis is that we’re trying to homogenize inherently heterogenous political events to make them tractable to statistical analysis. But presidential elections are not identical balls drawn randomly from an urn, and they are not lottery numbers randomly picked by a computer.

Paul October 19, 2012 at 9:48 am

I disagree with Andrew that 65.7% is too precise. One reason is that intrade prices are quoted to 3 digit precision — viz., ranging from $0.00 to $10.00, with bid-ask spreads as low as 1 cent. So, for example, if I believed Nate’s formula was more accurate than Intrade, and the current quote was $6.56 bid, $6.57 ask, I would like Nate to provide more precision in order to determine whether or not to buy ‘Barack Obama to be re-elected on 2012′. Even with today’s low interest rates, I would still need Nate to forecast at least a 65.74% probability in order to believe purchasing ‘Obama 2012′ at $6.57 would outperform a 0.90% money market rate over the next 18 days.

In addition, since this security has an all-or-nothing payout, I could use additional precision to determine, via the Kelly criterion, an optimal allocation of my portfolio.

Andrew Gelman October 19, 2012 at 11:33 am

Paul:

I can see why Intrade would want that precision. I just can’t see the point of Nate giving probabilities to that level of precision, given that he’s not working for Intrade or for a trader.

Andrew Gelman October 19, 2012 at 9:52 am

Jeffrey,

I recommend you take a look at chapter 1 of Bayesian Data Analysis, where we give a few examples of probabilities that are empirically determined from data. The different events being analyzed (in this case, elections) are not modeled as identical balls drawn from an urn. A better analogy you might keep in your mind is business forecasting. You might have some uncertainty about the price of oil next year, or whether the price will exceed $X a barrel. It’s an uncertain event but you know something about it. Then you get some information, for example a new oil field is discovered or a new refinery somewhere is built. This changes your probability. A number such as 65% could express this. Similarly, I might describe someone as being 5 feet 8 inches tall, or even 5 feet 8 1/2 inches tall, but it would be silly to call him 5 feed 8.34 inches tall, given that his height changes by a large fraction of an inch during the day.

John G October 19, 2012 at 2:56 pm

Bourdieu in Science of Science and Reflexivity provides a lot of insight to this post. Like, “the peer recognition that characterizes the field tends to produce an effect of closure” (56). I suspect that Nate is influential. There is bias in the numbers because there is undisclosed bias in the market to become President. And InTrade should minimize bias by comparing analysis. Selection bias may be salient, but if the population is strongly trending liberal (like the wave experienced in 2006), then swing states may all go Obama, which would create a 99.9% bet for Obama.

50percentmoreflavor October 19, 2012 at 10:14 pm

The dark side of the moon is more liquid than Intrade.

5 grand would move any maket they make more than 10 percent. There is no accuracy if there’s no liquidity.

This Nate Silver character is quite the carnival barker. He’s the worst sort of hustler because he chases hot trends. Baseball metrics: fail. Poker player: fail. Etc., etc.

Jeffrey Friedman October 20, 2012 at 1:26 pm

Andrew—Unfortunately, your book is checked out of the library. But my understanding of Bayesianism is that the work is largely (as always, arguably) done by the model. (Correct me if I’m wrong.)

So what is Silver’s model drawn from? Since it is able to generate such precise predictions, it must be derived by attaching precise weights to a variety of factors that are then combined in a formula. Where does Silver get these factors and their weights? Presumably they fit past election data.

I cannot find anything on FiveThirtyEight about its methodology for the presidential election, but when one clicks “methodology” one is led to a description of senatorial-race methodology that confirms my assumption above: See, e.g., “pollster ratings” and particularly “Step 3” at http://fivethirtyeight.blogs.nytimes.com/methodology/ –here is a selection from Step 3 listing factors that are weighted in the model:

A state’s Partisan Voting Index
The composition of party identification in the state’s electorate (as determined through Gallup polling)
The sum of individual contributions received by each candidate as of the last F.E.C. reporting period (this variable is omitted if one or both candidates are new to the race and have yet to complete an FEC filing period)
Incumbency status
For incumbent Senators, an average of recent approval and favorability ratings
A variable representing stature, based on the highest elected office that the candidate has held. It takes on the value of 3 for candidates who have been Senators or Governors in the past; 2 for U.S. Representatives, statewide officeholders like Attorneys General, and mayors of cities of at least 300,000 persons; 1 for state senators, state representatives, and other material elected officeholders (like county commissioners or mayors of small cities), and 0 for candidates who have not held a material elected office before.

So the premise of the model seems to be that there is a fixed relationship among certain factors that is constant across elections. Yet voters, being people, decide elections by interpreting circumstances in unpredictable ways. Will bad economic times be blamed on Obama, as they were blamed on Hoover, or will he be credited with doing his best, as FDR was? Will a candidate say something or run an ad that changes people’s minds in a particularly effective manner? Will a debate performance, a decision to suspend one’s campaign because of a financial crisis, or the allocation of persuasive resources to a state that is actually a lost cause have an unprecedented effect?

Such possibilities, which are drawn from recent history, could not have been predicted before they happened, because they depended ultimately on constellations of ideas about contemporary circumstances that are held (and shaped) by voters, candidates, media personnel, and campaign staffs responding to their perceptions of unique circumstances. (They were black swans, but these are frequently sighted in human affairs.) If there is some underlying law that would allow us to predict people’s thoughts and thus their behavior in as-yet-unknown circumstances, I’d like to hear how the law has been discovered except by question-beggingly deriving it from past relationships.

Andrew Gelman October 20, 2012 at 3:58 pm

Jeffrey:

You write, “the premise of the model seems to be that there is a fixed relationship among certain factors that is constant across elections.” I don’t know what Nate did, but in general it is possible to allow these relationships to vary. There is no underlying law, just some historical consistency, just as in the analogy of business forecasting that I mentioned above.

Peter T October 21, 2012 at 8:36 am

I recall Nate’s Australian counterpart, Possum Comitatus (blogs at http://blogs.crikey.com.au/pollytics/) looked closely at Intrade prices for Australian elections, and concluded that they were driven more by the polls than by independent estimates. In other words, the polls proved more accurate when they diverged. Apologies that I lack the time to look up the particular post.

Jeffrey Friedman October 21, 2012 at 10:33 am

Andrew–Business forecasts are notoriously unreliable. That is one reason so many businesses lose money or go broke.

Of course there is historical consistency, but in human affairs one cannot know in advance which factors will remain consistent in a given case, especially in precise quantitative terms.

John Girdwood October 22, 2012 at 8:26 pm

Bayesianism can account for multiple factors–and we can assume that Nate’s model does not account for all variables (but that doesn’t mean Nate won’t be accurate 95% of the time either). My main problem, as I read, is that big factors seem to be missing. For example, since we are talking about voting behavior–there doesn’t seem to be any account of referendums. This is important because some voters –median voters–are likely impacted along the “soft choices and hard choices” debate in the voting booth according to referendums. Referendums might provide a cue to voters about which side seems heavy or heavy handed–as so to vote against it. So if I was creating a model to predict the election in swing states (as proportional electoral counts are (the future for democracy) 96% rare), I would calculate predicted weight on vote turnout with referendums too. I would factor in voting patterns from the pool of already collected research (some posted in the Cage) regarding how median voters might be persuaded to choose one party over the other according to any referendum. Second, I see no account for partisan tides, which is important.

John Girdwood October 26, 2012 at 1:35 pm

I think that there is a larger discussion here: Partisans and Independents as they change over time, and how they change America’s direction. I think that there is a narrative which causes too much closure on the topic of vote choice–the idea that almost all voters are actually partisans. I fundamentally disagree. I briefly talk about it here: http://politicalpipeline.wordpress.com/2012/10/26/partisans-and-independents-in-vote-choice

Comments on this entry are closed.

Previous post:

Next post: