« Oh Those Pesky Filtered Questions | Main | Season's Greetings! »

December 24, 2004

Have the Exit Polls Been Wrong Before?

I have a short backlog of posts on the exit polls I've been working on this week, intended mostly to summarize information I've covered previously and make it more accessible via the FAQ. However, there is new information here, as well as in the posts that will follow.

One of the odd bits of received wisdom I keep hearing about the exit poll controversy is that up until this year, the exit polls were "always right." If so then this year's errors seem "implausible," and wild conspiracy theories of a widespread fraud in the count somehow seem more credible. The problem with this reasoning is that exit polls similarly "wrong" before, though perhaps not to the same degree or consistency.

Here is the documentation on previous errors. First, from the Washington Post's Richard Morin:

The networks' 1992 national exit poll overstated Democrat Bill Clinton's advantage by 2.5 percentage points, about the same as the Kerry skew

Warren Mitofsky, who ran the 2004 exit poll operation along with partner Joe Lenski, wrote the following in the Spring 2003 issue of Public Opinion Quarterly (p. 51):

An inspection of within-precinct error in the exit poll for senate and governor races in 1990, 1994 and 1998 shows an understatement of the Democratic candidate for 20 percent of the 180 polls in that time period and an overstatement 38 percent of the time...the most likely source of this error is differential non-response rates for Democrats and Republicans:

From the internal CNN report on the network's performance on Election Night 2000 (p. 48 of pdf):

Warren Mitofsky and Joe Lenski, heads of the CNN/CBS Decision Team, told us in our January 26 interview with them that in VNS's use of exit polls on Election Day 2000, the exit polls overstated the Gore vote in 22 states and overstated the Bush vote in 9 states. In the other 19 states, the polls matched actual results. There was a similar Democratic candidate overstatement in 1996 and a larger one in 1992.

In short, Mitofsky and Lenski have reported Democratic overstatements to some degree in every election since 1990. Moreover, all of Lenski and Mitofsky's statements were on the record long before Election Day 2004.

Of course, those errors were apparently bigger and more consistent this year. According to an internal NEP report leaked to the New York Times, this year's "surveys had the biggest partisan skew since at least 1988, the earliest election the report tracked." However, in some states, the errors in 2000 were still quite large. Consider this comment from Joe Lenski to CNN on December 12, 2000 (p. 48 of pdf), describing the table also copied below: 

The second group contains five states that had stupendously bad exit poll estimates. Here is a comparison of the final best survey estimate at poll closing with the final actual results for these five states... As you can see the exit polls in these five states were off by between 7 and 16(!!!) [Emphasis in original]

Lenski_table

The exit poll errors four years ago led Mitofsky to tell the CNN investigators, "The exit poll is a blunt instrument," and Lenski to add, "the polls are getting less accurate" (p. 26 of pdf). They recommended "raising the bar" on projections made from exit polls: "The proposed changes result from a belief that exit polling is "less accurate than it was before" and that "we should take exit poll data with caution in making calls," said Lenski" (p. 27).

All of this led the authors of the internal CNN report -- Joan Konner, James Risser, and Ben Wattenberg - to conclude (p. 3, 7):

Exit polling is extremely valuable as a source of post-election information about the electorate. But it has lost much of the value it had for projecting election results in close elections...[Their recommendation to CNN:] Cease the use of exit polling to project or call winners of states. The 2000 election demonstrates the faults and dangers in exit polling. Even if exit polling is made more accurate, it will never be as accurate as a properly conducted actual vote count.

[FAQ on Exit Polls]

Related Entries - Exit Polls

Posted by Mark Blumenthal on December 24, 2004 at 10:23 PM in Exit Polls | Permalink

Comments

Merry Christmas, Mark, to you and yours, and a thanks for all the great info you've provided. It's been enormously instructive and is highly valued here.

Posted by: Kevin Hayden | Dec 25, 2004 4:40:37 AM

Great Stuff Mark and enjoy your vacation. I've actually been at the library a few times this week digging up everything I can on the historical accuracy of the exit polls. There are very few books and journal articles (you have hit on almost all of them at some place on this blog), but I've moved into newspaper articles written in the days and weeks following elections since 1980 (presidential and non-presidential). It's really quite fascinating! Thanks for additional sources here. Best wishes this holiday season and I look forward to your forthcoming posts on this matter.

Posted by: Rick Brady | Dec 25, 2004 9:00:28 PM

Also, Mark, I hope you address this, but how many battleground states were "called" based on the exit polls? As I'm learning, there is a big difference between the exit poll data circulated for analysis to the media and the data used to project winners.

Posted by: Rick Brady | Dec 25, 2004 10:52:16 PM

Scratch that... difference in the methods and analysis, not data...

Posted by: Rick | Dec 25, 2004 11:05:27 PM

Thank you Mark. Holiday greetings to you and your family.

I appreciate reading your work on setting a historical context for exit polling.

What is most interesting for me is learning how much of a creaking election system we have in the US. No professional election commission, horribly inconsistent election procedures, hardly a thought given or effort made in systematically auditing election results, and exit polling treated as "electioneering" at too many precincts so that compared to Germany if not Europe exit polls are sorely lacking.

Most telling is how much of this is a complete novelty because it has gone unnoticed in the past (probably for decades). Is it just me or does it seem like the nation has been asleep at the wheel? Hard to comprehend frankly.

Posted by: Alex in Los Angeles | Dec 27, 2004 10:23:59 PM

The comments to this entry are closed.