Models, Models, Everywhere

By NATE SILVER

We assign a lot of bandwidth to the FiveThirtyEight presidential forecast model, but that doesn't mean it's the only thing you should look at to get a sense for where the election is headed. So it's time for a quick gut-check. How does the FiveThirtyEight model - where Barack Obama's probability of winning the Electoral College recently ticked above 70 percent - compare against alternative means of forecasting the election?

There are a number of other statistical forecasting systems, most of which rely on polls, economic variables or some combination thereof.

I tracked down about every one of these models that I could find, subject to the condition that it couched its forecast in probabilistic terms (or that it was well-documented enough to allow this to be inferred with relative ease, like from the standard error that the model stated).

In my view, it's in estimating the uncertainty in a forecast where most of the challenge and intrigue lies. To paraphrase Charles Barkley, any knucklehead can make a point prediction - but it takes brains to calculate a confidence interval.

These models are all over the map, forecasting everything from a nearly certain Obama victory to the substantial likelihood of his defeat. But more of them have Mr. Obama as the favorite. If you simply average their win probability estimates together, you get about a 61 percent likelihood of his winning the election.

This calculation combines models that estimate Mr. Obama's chances of winning the popular vote (there are marked with “P.V.” in the chart) and those that make an Electoral College forecast. I list the FiveThirtyEight forecast twice since it provides both estimates.

The methods that make an Electoral College forecast are rather bullish on Mr. Obama - but be a little careful about reading too much from that. I do think Mr. Obama has a modest advantage i n the Electoral College relative to the popular vote. But it isn't more than modest: Mr. Obama is almost certain to lose the Electoral College if he loses the popular vote by more than a percentage point or two. Most of the models that make popular vote forecasts are more driven by economic factors, while those that make Electoral College picks rely more on polls, so they are not necessarily comparable.

The table also contains a couple of comparatively simple methods for forecasting the election. One, which I call “just G.D.P.,” is derived from looking at the growth rate of real gross domestic product in the first three economic quarters of election years since 1948, and comparing it with the incumbent party's margin in the popular vote. As its name implies, the “just G.D.P.” method does not include any other variables apart from G.D.P.

Its companion, “just jobs,” looks at the growth rate of nonfarm payrolls in January through October of the election year, with no other variables or adjustments. For “just G.D.P.” and “just jobs,” I use consensus forecasts from The Wall Street Journal's forecasting panel to fill in the economic numbers for the balance of 2012.

These simple methods predict a win for Mr. Obama - but with a lot of hesitation, by about 1 percentage point in the national popular vote. That translates to about a 55 percent chance of a popular vote victory, given the standard error in the models. It's not precisely even, but close enough that poker players would call it a “coin flip.”

As for the rest of the models - as longtime readers will know, I've sometimes had harsh things to say about them. But really, I think they fall into about four different categories:

1) Models that are basically well designed.

2) Models that fall into the category of “if you can't say anything nice, don't get into a flame war on Twitter about them.”

3) Models that are overly simplistic, but that form useful benchmarks precisely because of their simplicity. (“Just jobs” and “just G.D.P.” fall into this category, along with some of the others.)

4) Models that provide sensible projections of the margin between the candidates, and which are useful for that purpose, but which don't do a very good job of translating that into probabilities.

What's interesting, however - and perhaps not entirely coincidental - is that if you do take the average estimate from these models (Mr. Obama with just slightly better than a 60 percent chance of victory as of late Monday night), it almost exactly matches the consensus of gambling odds from different betting markets and bookmakers.

The second chart, below, provides a list of these. I looked up the odds for Mr. Obama and Mitt Romney at various futures markets, and at different sports books.

For this chart, I've provided two sets of numbers: the columns labeled “raw odd s” list the numbers as-is from the betting sites, while the columns labeled “adjusted odds” tweak them so that the collective win probabilities for Mr. Obama and Mr. Romney add up to exactly 100 percent. (Why don't the numbers add up to 100 percent to begin with? In the case of the futures markets, because they assign a small probability to candidates like Ron Paul or Hillary Rodham Clinton winning the election; for the sports books, because the unadjusted odds provide for a cut to the betting house and are intrinsically slightly disadvantageous to the bettor.)

The sports books are a smidgen more bullish on Mr. Obama than the betting markets. But both gravitate to a win probability of about 61 or 62 percent on an adjusted basis, just as the average of statistical methods do.

You do see that these markets aren't terribly efficient. Mr. Obama's win percentage is about four percentage points higher at Betfair than at Intrade, for instance, the two most liquid futures markets. And various sports books peg his odds anywhere from 59 percent to 68 percent, on an adjusted basis. In theory, there could be arbitrage opportunities based on the differences between these numbers.

The FiveThirtyEight model was quite close to the consensus until recently, but has now become a bit more bullish on Mr. Obama's chances, listing his Electoral College odds at slightly over 70 percent - instead of slightly over 60 percent, as the consensus does.

I sometimes get asked whether I bet money on my forecasts - I don't, since I would consider it a conflict of interest - or failing that, whether I would recommend a bet on them relative to the odds on offer at Intrade or Betfair.

My answer is probably unsatisfying. I think modeling a presidential election is a pretty hard problem. I think futures markets and sports books (like markets of any kind) can certainly go wrong. But I also think that the statistical methods can go wrong: all of th em rely on a set of assumptions and choices made by the forecaster.

Some choices, in my view, are clearly better than others. One or two of the statistical methods, for instance, assumes that the outcome in each state is independent of the outcome in the next one. Ohio might move in one direction - and Michigan, just as easily, in the opposite one.

That's simply not a credible assumption. The failure to appreciate correlations in risk is one of the things that led to the recent financial crisis. A change in economic conditions, or a substantial gaffe or scandal in the campaign, is likely to be reflected to some degree in all states, and move all of their numbers in the same direction. Our model assumes that the uncertainty in different states is largely, but not entirely, correlated. If you believe the contrary, you probably ought not be let anywhere near a job function in which you are asked to manage risk - although the credit-ratings agencies might be happy to hire you.

These pet peeves aside, elections forecasting is a challenging problem. More often, the assumptions in a model are intrinsically going to be educated guesses rather than being demonstrably right or wrong.

So my default is this: Bet on Vegas relative to the FiveThirtyEight model, but bet on the FiveThirtyEight model relative to Vegas. If you take the average between the FiveThirtyEight model and the consensus betting lines, you'd get about a two-in-three chance of Mr. Obama winning another term.