Exits: Were They Really “Wrong?”

Exit Polls Legacy blog posts

Last week’s posting of more detailed information on the sampling error of exit polls by the National Election Pool (NEP) allows for a quick review of the now well established conventional wisdom that the “exit polls were wrong.” 

Let’s first set aside the mid-day numbers that were widely leaked on the Internet but never intended as the basis for projections (numbers that even the exit pollsters recognized as flawed in some states — see this earlier post for explanations on the differences among the many estimates provided by the exit pollsters on Election Day). Let us also stipulate that at least one state estimate of Hispanic voters (Texas) was obviously wrong, given the correction issued by NEP.

The conclusion that the exit polls were wrong is essentially about two possible failings:

  • That the end-of-day numbers seemed to predict a Kerry victory.
  • That the end-of-day numbers showed Kerry doing consistently better than he actually did.

What is the reality given what we now know about the exit poll’s sampling error?

1) Did the just-before-poll-closing exit polls show a consistent and statistically significant “error” in Kerry’s favor?

Yes, but that error has been exaggerated. Here is what we know:

  • An internal NEP review of 1,400 sample precincts showed Kerry’s share of the vote overstated by an average of 1.9 percentage points. As far as I can tell, no one from NEP questions the statistical significance of that overstatement.
  • The before-poll-closing exit poll results posted by Steven Freeman (and included in the updated report by the Cal Tech / MIT Vote Project) show errors in Kerry’s favor in 43 of the 51 surveys (the states plus DC). These overstate Kerry’s vote by an average of 1.6 percentage points. If the surveys had been perfectly executed with perfectly random samples (an all but impossible assumption under real world conditions), the pattern of errors should have been the same as if we had flipped a coin 51 times: about half favoring Kerry, about half favoring Bush. The probability of 43 of 51 perfect surveys favoring Kerry by chance alone is less than 0.001%. Of course, this statistic only tells us that the surveys were imperfect. It says nothing about the cause or magnitude of the error.

To be clear: Everyone — including the exit pollsters — agrees they “overstated” Kerry’s vote. There is some argument about the precise degree of certainty of that overstatement, but if all agree that the difference is statistically significant the degree of certainty has little consequence. The size of the error matters, and the reasons for it matter, but whether our level of confidence about the error’s existence is 99.9% or something greater does not.

Having said that, the second draft of the paper, “The Unexplained Exit Poll Discrepancy,” continues to needlessly exaggerate the significance of the error, especially within individual states. For example, Freeman claims that there were significant “discrepancies” in Pennsylvania, Ohio and Florida when each state is considered separately. He has a graphic (Figure 1.2) showing a significant error in Florida, assuming a 95% confidence level. However, these assertions are not supported by the margins of error reported by NEP.

  • I applied the appropriate “confidence intervals” reported by NEP (as distributed to its partner networks on or before Election Day) to each state. Contrary to Freeman’s assertions, the separate “discrepancies” in Ohio, Florida and Pennsylvania fail to attain statistical significance even at a 95% confidence level. In fact, I see only four states (New Hampshire, New York, South Carolina and Vermont) with statistically significant errors favoring Kerry at a 95% confidence level. Of course, when using a 95% confidence level, 2-3 states should be out of range by chance alone.
  • I also followed the advice of Nick Panagakis and estimated confidence intervals at a 99.5% level of confidence (the standard used by NEP to make projections) using the actual estimates of the design effect obtained from NEP Warren Mitofsky by blogger Rick Brady. By my calculations, this more demanding test renders the apparent errors in NH, NY, SC and VT non-significant.

Then there is the statistic heard round the world – Freeman’s original “250 million to one” estimate of the odds against the discrepancy occurring simultaneously in Pennsylvania, Ohio and Florida. In his second draft, Freeman revised his estimate down to a mere 662,000 to one. Given that Freeman continues to understate the “design effect” used to calculate the sampling error in this year’s exit polls, his revised estimate also remains too high.

Some have asked that I calculate my own estimate of the joint probability of an error in Ohio, Florida and Pennsylvania. I am reluctant to do so for two reasons: First, the rounding error in Freeman’s data alone renders this sort of hairsplitting moot. Second, and more important, it really doesn’t matter. Everyone concedes there was a small (2%) but significant average error in Kerry’s direction. For those concerned about problems with the count, what matters most is why that error occurred.

2) Considering sampling error, did the end-of-day numbers predict a Kerry victory?

This one is easy: No. Not even close.

It is true that the early numbers fooled a lot of people, including the talking heads on the cable networks, pollsters and advisors of both campaigns and possibly — depending on which accounts you believe — even John Kerry and George Bush. The reason, as the Post‘s Richard Morin put it, is that the 1.9% average error in Kerry’s favor “was just enough to create an entirely wrong impression about the direction of the race in a number of key states and nationally.”

True. But that error was not big enough to give Kerry statistically significant leads in enough states to indicate a Kerry victory. If we totally ignore sampling error, the exit polls showed Kerry ahead in only four states that he ultimately lost:  Ohio, Iowa, Nevada and New Mexico. Obviously, Kerry would have won the election had he prevailed in all four.

Before considering whether any of those leads were statistically significant, two quick reminders: First, as Nick Panagakis points out, NEP required at least a 99% confidence level for projections on Election Night. Second, the margin between two candidates on a survey involves two estimates; and the margin of error applies separately to each candidate. Statisticians debate which rules of thumb to apply when determining the significance of the margin between two candidates, but the consensus in the case falls somewhere between the 1.7, as recommended by the American Statistical Association and 2.0, as recommended by those who consider it more appropriate in a race where the vote for 3rd party candidates is negligible (this is a great topic for another day’s post – thanks to alert reader Bill Kaminsky for guiding MP through the competing arguments).

Fortunately, in this case, the statistical debate is irrelevant. None of Kerry’s apparent exit poll leads in the four states were large enough to attain statistical significance, even if we assume a 95% confidence level and use a cautious multiplier (1.7) on the margin of error. As the preceding table shows, the exit polls had Kerry ahead by 4 percentage points in Ohio, by 3 in New Mexico, by 2 in Iowa and by one point in Nevada. The NEP 95% margin of error for these states multiplied by 1.7 is between 5 and 7 percentage points. At the more appropriate confidence level of 99.5% — the one NEP actually uses to recommend projections – these relatively small margins would fall far short of that needed to call a winner.

In a year-end review in this week’s Roll Call (subscription required), Stuart Rothenberg concluded the following:

The problem wasn’t the exits – it was the folks who treated the early-afternoon numbers as if they were a predictor of what would happen after everyone had voted. The exit poll was off by a couple of points, but that’s well within the margin of error.

If only someone had warned us about those early afternoon numbers on Election Day. Oh, wait

UPDATE:  I mentioned William Kaminski’s blog, but not his commentary on this issue nor his elegant graphic — it’s worth the click just for the chart.  My only remaining quibble with his graphic is that it displays a “safe estimate” of a 95% confidence of +/-4%; leaving the impression that the numbers for DE, AL, AK and NE fall outside that interval.  However, the appropriate 95% confidence interval provided by NEP for these states is +/-5%.

[FAQ on Exit Polls]

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.