Aug. 23: Seven Ways to Evaluate a Poll

By NATE SILVER

Each night before I run our presidential forecast model, I make a guess at what it will say based on my impressionistic take on the data from that day.

I figured that Thursday would be something of a push. Barack Obama held the lead in a number of state polls. But the numbers were not necessarily better for him than the prior renditions of the same surveys. Also, Mitt Romney got a strong number in the Fox News national survey - and the stock market declined fairly significantly, which lowered the model's economic index.

The conclusion of the model was less equivocal. It evaluated the state polls as being quite strong for Mr. Obama, and they outweighed the other factors. His chances of winning the Electoral College rose to 68.6 percent in the forecast from 66.7 percent on Wednesday.

Mr. Obama saw the most improvement to his forecast in Ohio, where his winning chances improved to 70 percent from 64 percent. In that s tate, there was a new survey from Quinnipiac, which was conducted in conjunction with the New York Times and CBS News. It gave Mr. Obama a 6-point lead there. There was also a survey from the University of Cincinnati, which put him ahead by three points.

Let's focus on that Quinnipiac poll, however. It showed Mr. Obama with a six-point lead. But Mr. Obama had a lead by the same margin in its July poll. So why does the model evaluate it as a significant positive for him?

The reason is that looking at changes in poll numbers is a useful habit, but hardly the only comparison you should be making.

Instead, there are at least seven ways that you might evaluate a poll as being strong or weak for a candidate. In the case of the Ohio poll, the comparison to the July survey showed no net change toward either candidate - but the poll was nevertheless strong for Mr. Obama by several of these measures.

1. How does the poll compare to other recent surveys of the s tate? This is a simple one, and in the Quinnipiac poll, this comparison was favorable for Mr. Obama. He led by six points, better than his average lead of one point in the other Ohio surveys conducted so far in August.

2. How does the poll compare to the polling firm's previous surveys in the state? Note my usage of the plural, surveys. You should be looking at how a new poll compares to the prior one that a survey firm released in a state - but also in comparison to its entire body of work there.

For example, suppose that a polling firm conducts a survey of Virginia every month. In February, March, April, May and June, the survey shows a small but consistent lead for Mr. Romney of between one and three percentage points.

Then, suddenly, Mr. Romney's lead increases to 10 points in the July poll.

What would qualify as a good number for Mr. Romney when the firm releases its August survey?

If the August poll put Mr. Romney eight points ahead, that would still count as a very strong number for him. It's down a bit from the July poll, but much better than he'd gotten in the other months of the year.

When a poll initially appears to be an outlier, you should expect there to be some reversion to the mean the next time the firm takes the temperature of the state. In these cases, just holding steady in a poll can count as a big “win” for a candidate, since what once looked like an outlier now looks like more of a trend. Even falling by a point or two can be a pretty decent result under some circumstances.

The July Quinnipiac poll in Ohio, which gave Mr. Obama a six-point lead, was not quite an outlier; Mr. Obama has periodically gotten numbers like that in Ohio throughout the campaign. Still, it was better for Mr. Obama than the surveys that Quinnipiac had conducted in Ohio prior to July, which put him up by an average of three points. (Moreover, the polls Quinnipiac conducted prior to July had been conduct ed among registered voters rather than likely voters. Usually, the shift to likely voters harms the Democratic candidate - but in this poll Mr. Obama's numbers have improved instead.)

3. How does the survey compare with the polling firm's surveys in other states? A related question is whether the polling has a strong “house effect.” That is, does it consistently show greater numbers for a particular candidate, no matter where it polls? If so, then it's not really a big deal when it ventures into a new state and print another one.

Or, does it show strong numbers for a candidate in some states, but not others? This is more of the case with Quinnipiac. It has had good numbers for Mr. Obama in Ohio, Florida and Pennsylvania recently - but not as much in the other swing states that it polls, or in noncompetitive states like New York or in its national surveys. In fact, the model calculates Quinnipiac's house effect as being just a tiny bit Republican-leaning on ba lance. That makes its poll showing a solid lead for Mr. Obama in Ohio more meaningful.

4. How does the poll compare with the national trend? This is the trickiest of the seven concepts, but one of the more important ones. When comparing a poll with prior surveys of the state, it's important to look at exactly when each survey was conducted, and what the national environment looked like at the time.

Say a polling firm comes out with a new survey in Michigan, showing Mr. Obama seven points ahead there. The firm's lone prior poll of the state had been in February, when it also had Mr. Obama seven points up.

This would count as a favorable development for Mr. Obama. Why? Because February was a very strong month for Mr. Obama in the polls. (At the time, Mr. Romney's numbers were really suffering because of his problems in the Republican primaries.) You'd have anticipated some decline for Mr. Obama since then; if his numbers held steady instead, that would defy your expectations and would be a constitute a positive sign for him.

Likewise, the last couple weeks have shown stronger numbers for Mr. Romney than he got in July. (Perhaps this was because of a small bounce for Mr. Romney following the selection of Representative Paul D. Ryan as his running mate.) That is priced into the model at this point. When a new survey comes out, its expectation is that the numbers will be a point or so stronger for Mr. Romney than they were a month ago - and it will be disappointed in Mr. Romney if he does not match them.

The state polls that Mr. Romney got on Thursday were not terrible by any means. But they looked more like the numbers we were seeing in June and July - when Mr. Obama led in the vast majority of swing state polls - and less like what we were seeing over the past few weeks, when Mr. Romney was leading in almost half of the polls in states like these.

A side note: Be careful of making comparisons to polls conducted immediately after the party conventions, when the candidates can experience significant (but short-lived) bounces. Say that Mr. Romney holds a seven-point lead in a North Carolina poll conducted in the immediate aftermath of next week's Republican National Convention, a time when Mr. Romney might be enjoying a significant bounce in the polls. The polling firm surveys the state again in late September, and Mr. Romney's lead is down to one point.

When something like this happens, you'll see blogs with headlines like “Romney Collapsing in North Carolina!” You should avoid taking these very seriously. All that happened is that you are comparing a poll conducted in the midst of Mr. Romney's convention bounce to one that wasn't. It is unlikely that the change tells you anything about North Carolina in particular.

5. How does the poll compare with the historical trend in a state? A state poll can be used to make an inference about where the national race stands. Oh io, for example, is typically very slightly Republican-leaning relative to the country as a whole. So when Mr. Obama gets a poll showing him 6 points ahead there, that speaks favorably to his position in the national race.

A related point is that the states often revert to the mean over the course of an election cycle. If a candidate seems to be overperforming in a particular state - as Mr. Obama has been in Ohio - there can be downward pressure on his numbers as the state's partisan gravity kicks in.

Our forecast model accounts for this property. It is designed to be a little skeptical that, for instance, Mr. Obama is actually leading by a larger margin in Ohio (by about three points in the polls) than in the national numbers (where he leads by about two points). However, the longer a candidate is able to defy gravity, the more confidence we might have that there is something different about the state this year.

6. How does the poll relate to the electora l calendar? Any time that a candidate is leading in a state, and he simply maintains that lead, that counts as a modest positive for him because time is running off his opponent's clock. A three-point deficit in April is very easy to overcome. In August? Still plenty of time, but there's room for a touch of concern. On election eve? Oops - it's probably too late.

There's no need for Mr. Romney to panic yet, especially since he has gotten his share of decent state polls recently. But if Mr. Obama still holds a three-point lead in Ohio after the party conventions, it's going to qualify as a negative for Mr. Romney every time that he fails to reduce the gap.

7. How does the number of undecided voters compare with prior renditions of the survey? In general, I focus on the margin between the candidates (“Mr. Obama's up by three points”) rather than their absolute numbers (“Mr. Obama 47 percent, Mr. Romney 44 percent”), or the number of undecided voters in the state.

This is partly because there are a lot of myths about how undecided voters behave, and I don't want to encourage those.

But the main reason is that, just as there are house effects related to whether a polling firm leans Republican or Democratic, different polling firms also consistently show more or fewer undecided voters, depending on how much the survey questions “push” voters toward a candidate choice. Our model actually adjusts for this, but it's something that can escape your attention unless you're being super detail-oriented.

Nevertheless, there is some useful information in the number of undecided voters in a race - especially when you are making comparisons to the prior editions of the same survey.

As a rule, the fewer undecided voters in a state, the less uncertainty there is about the eventual outcome. Thus, the candidate who is leading in a state should be pleased when the number of undecided voters falls, as this can make a s mall lead more robust. A 49-47 deficit is harder to overcome than 45-43, even though the margin between the candidates is two points in each instance.

***

Isn't this an awful lot to keep track of? It certainly is, and that's why a computer program can come in handy. The model can and does evaluate all this context when it considers a poll, and sometimes it concludes that a set of survey results are stronger or weaker for a candidate than they might appear on the surface.

Then again, the story of Thursday's polls wasn't all that complicated: Mr. Obama held leads in all six of the polls released of voters in Ohio, Florida, Pennsylvania, Wisconsin and Michigan - and Mr. Romney will need to win at least two of those states, and possibly three, to have much of a chance in the Electoral College. Mr. Romney has had some strong polling days recently, but this was not one of his better ones.