Tuesday, August 21, 2012

Aug. 20: When the Polling Gets Weird

By NATE SILVER

Monday brought a Monmouth University poll showing President Obama ahead by one point in the national race - the same margin that he had in the last poll they conducted in June. Mitt Romney held onto a 2-point lead in the Gallup national tracking poll, and pulled ahead of Mr. Obama again by one point in the Rasmussen national tracker, which has fluctuated a bit recently but usually has a steady number like that.

Oh, and there was an Oklahoma poll, which was conducted on behalf of the Tulsa World. Mr. Obama is surging there! He now trails Mr. Romney by only 29 points. (He was 35 points down in the prior rendition of the same survey.)

All of this polling was pretty normal, of course. But there was one last survey. It was from the polling firm Foster McCollum White Baydoun, which conducts polls for Democratic candidates as well as independently. It was a poll of Florida and it had Mr. Romney ahead by nearly 15 points there.< /p>

A general word about Florida: I'm mostly on board with this post by Nate Cohn, which criticizes the idea that Paul D. Ryan could be especially damaging to Mr. Romney's chances there. (The theory Mr. Cohn is arguing against is that there will be some sort of revolt among seniors in Florida against Mr. Ryan's Medicare plan.)

The basic flaw in the argument, as Mr. Cohn notes, is this: there are a lot of seniors who will vote in Florida. But there are also a lot of seniors who will vote in every state. Age is not one of those demographics that varies that widely in different parts of the country.

In 2008, for instance, voters 65 and older made up 22 percent of Florida's electorate, according to exit polls there. By contrast, the same demographic made up 16 percent of the national vote.

Imagine that, hypothetically, the choice of Mr. Ryan cost Mr. Romney a net of five points among voters in this age group. If you do the math, t his means that the Republicans would lose a net of 1.1 percentage points in Florida, and a net of 0.8 percentage points elsewhere in the country. We all know how much every vote can matter in Florida, but a 0.3-point swing in Florida relative to the national trend is just not enough to change the electoral math by that much. In the universe where Mr. Ryan does significant damage to Mr. Romney among seniors, the Republicans aren't going to have a Florida problem - they're going to have problems everywhere else on the map.

Even if Mr. Ryan's proposals on entitlement programs are damaging to Mr. Romney, it is not clear if the effects would be most acute among voters 65 and older. Mr. Ryan's Medicare plan would leave voters 55 and older exempt from most of its proposed changes, so if voters were behaving in a purely self-interested way, it would theoretically be a voter at age 54 (or somewhere else in late middle age) who would have the most to lose from the changes to the program, having paid the most into the system.

Obviously, the politics are not quite that simple - Democrats could campaign on the issue, and maybe their arguments would resonate among voters who would see less personal impact from Mr. Ryan's proposed changes. But at the very least, the theory that Mr. Ryan will be especially damaging to Mr. Romney in Florida is speculative.

With that said, I very much doubt that Mr. Romney and Mr. Ryan will win Florida by 15 percentage points, as the Foster McCollum White Baydoun poll currently says. The survey is a huge outlier relative to the consensus of polls in Florida, which have been a bit variable but have pointed toward a race that is roughly tied.

Regular readers will know that I generally refrain from making comments on the demographics within an individual poll. They will vary a bit from survey to survey based both on random variance and the different assumptions that pollsters make about just who will turn ou t. The random variance can be reduced by taking an average of surveys - and if pollsters have a persistent tendency to favor one candidate over another, we can account for that with our house effects adjustment, which is designed to detect and counteract these tendencies.

But once in a great while, a poll that comes along with methodology that is so implausible that it deserves some further comment. The Foster McCollum White Baydoun poll of Florida is one such survey.

The poll was weighted to a demographic estimate that predicts that just 2 percent of Florida voters will be 30 or younger. It's a decent bet that turnout will be down some among younger voters this year, but that isn't a realistic estimate. In 2008, according to exit polls, 15 percent of voters in Florida were between 18 and 30.

The poll also assumed that 10 percent of voters will be between the ages of 31 and 50. In 2008, the actual percentage was 36 percent, according to the exit survey.

The poll projected Latinos to be 7 percent of the turnout in Florida, against 14 percent in 2008. And it has African-American turnout at 10 percent, down from 13 percent.

If the turnout numbers look something like that in November, then Mr. Obama will lose Florida badly. He'll also lose almost every other state; his electoral map might look a lot like Walter Mondale's.

But the share of voters 50 and younger in Florida is not going to drop all the way from half the electorate to roughly one-tenth of it, as the poll assumed. That is far beyond the range you can get from reasonable disagreement about methods, or from sampling error. It looks like the result from a from a badly-designed statistical model that never got a sanity check.

With all that said, we still include the poll in our forecast model. (I've noticed that some other polling aggregation Web sites have not listed the Foster McCollum White Baydoun survey, although I'm not sure if that's an inten tional or unintentional oversight.)

It's probably a defensible call either way, but our view is that we'd rather deal with “weird” polls in a systemic way rather than having to make an editorial judgment about them.

For instance, we have our house effects adjustment, which corrects for most of these tendencies. Based on this poll, and a prior survey the firm conducted in Michigan, we calculate the firm's house effect as leaning Republican by roughly 11 percentage points relative to the overall consensus.

We do not subtract out the entire 11-point house effect from the polling firm's results - the model allows polling firms to retain some of their house effect - but the model does adjust the poll substantially, treating it as about a 7-point lead for Mr. Romney rather than a 15-point one. That's still a very good number for Mr. Romney - enough to make him a slight favorite in our forecast for the state - but at least a little bit more reasonable relative to common sense.

Is there argument for just throwing the poll out? In this case, perhaps. But as I said, I'd rather design a system where we have to make fewer of those judgment calls and err on the side of inclusivity. Our threshold for calling out a poll's technique as being dubious, as we have here, is pretty high - but our threshold for actually throwing a poll out is higher.



No comments:

Post a Comment