November 05, 2004
Exit Polls: The NEP Report
A story by Jim Rutenberg in today's New York Times finally reports the explanation of the exit poll mishap from the people who ran it, Warren Mitofsky and Joe Lenski of Mitofsky International & Edison Research. Rutenberg "obtained" a post-election report written by Mitofsky and Lenski for the subscribers of the exit poll conducted by the National Election Pool (NEP): the major networks and newspapers like the New York Times.
The Rutenberg piece is worth reading in full, but before doing so, consider again the different deliverables that NEP provided on Tuesday. First are the partial results reported in the middle of the afternoon and widely leaked on the Internet that showed Kerry doing a few points better in nearly every state poll than he did in the final result. Second are the numbers delivered for each state just before the polls closed, data that are typically weighted to match the day's actual turnout and that facilitate election projections. Finally, NEP provides final data well after the polls close that they weight by the vote to match actual results. These data are not used for projections but for analysis and subgroup comparisons, so weighting by the actual vote helps make the final poll more accurate.
One question I have been asking here is whether the second deliverable - the data delivered for each state as the polls closed - had the same Kerry bias as the early reports. The Rutenberg piece implies that it did:
According to the report, the surveys had the biggest partisan skew since at least 1988, the earliest election the report tracked.
"We share all the members' concerns about the inaccuracies in the projections produced by the early waves of exit poll data and we are personally miffed about the early results,'' the report said.
Rutenberg goes on to confirm that the skew in the national survey persisted until the end of the day:
"The last wave of national exit polls we received, along with many other subscribers, showed Kerry winning the popular vote by 51 percent to 48 percent, if true, surely enough to carry the Electoral College,'' Steve Coll, managing editor of The Washington Post, wrote in an online chat with readers Wednesday [emphasis added].
In an interview yesterday, Mr. Coll said his newspaper had to scramble to make last-minute changes to an article analyzing why voters voted the way they did that was based in part on the poll data when it was clear that no such victory for Mr. Kerry was possible.
"We think it wasn't worth what we paid for it, that's for sure,'' Mr. Coll said of the survey data.
What reasons did the NEP officials offer for the skew?
In some cases, the [NEP] report said, survey takers could not get close enough to the polls to collect adequate samples of voters opinion. They were often stopped by legal barriers devised to keep people electioneering - not necessarily bona fide poll canvassers - away from voters.
The report also theorized that the poll results more frequently overstated support for Mr. Kerry than for President Bush because the Democratic nominee's supporters were more open to pollsters.
Keep in mind that the exit pollsters can compare their results with reality all the way down to the precinct level. Speculation by NEP officials about a systematic non-response bias favoring Kerry is significant, because they are certainly in a position to confirm any such bias. Note also that Rutenberg's story makes no mention of the earlier speculation that early voting by Democrats caused a mid-day skew that faded later in the day. That theory would explain a temporary problem observed only in mid-day samples, but not a bias that persisted until the polls closed.
So armed with this new information, let me try to review some of the questions I have received from readers:
Q: Are we SURE that we can eliminate the possibility that some kind of electoral fraud tipped votes from the Democratic to Republican columns? Is it possible that the exit poll was right and the vote count was wrong?
The Rutenberg article says the NEP report "debunked" the emerging conspiracy theories that "the exit polls are right and the vote count is wrong." The article did not elaborate. However, since 2000, the exit pollsters have tracked the type of voting equipment used at their sampled precincts. If the discrepancies could be explained, as some suggest, by precincts using the newer Diebold touch-screen voting machines, the exit pollsters could prove it. With their own reputations on the line, the NEP officials report no such evidence.
Q: Was there evidence of "a deliberate act of fraud and bias" on the part of exit pollsters, as Dick Morris implies, or that Democrats "had an election-day project of slamming the results" as Mike Barone argues?
It is hard to take either suggestion seriously. The point of fraud by the exit pollsters would be what exactly? Putting their reputations at risk by deceiving their clients and the world just six hours before the real results are available? It makes no sense. And while I can imagine a campaign operative emailing fictional exit poll numbers to a blogger (something I suggested in point five of my "what you should know" post), the idea of a widespread campaign to manipulate the exit polls with pseudo-respondents is ludicrous. Again, what is the point? Were than many marginal voters really sitting by the computer deciding whether to vote on the basis of leaked exit polls? If Democrats were capable of pulling off such a massive, secret conspiracy to pad the exit polls by a point or two in every state, why not just put the same effort into mobilizing real voters?
Q: "Did the early exit polls showing Kerry ahead almost across the board actually spur pro-Bush voters to head to the polls?"
Sorry Mickey, but I doubt it. To know for sure, it would help to know how many voters were even aware on Election Day that Kerry led in early exit polls. I would be stunned if the number is greater than a few million, and virtually all were hard-core political junkies already certain to vote. The blogosphere has great reach, but not that great.
Q: How can we be sure that the post-election analysis from the exit polls is accurate?
This is the most important question of all. Since the final results are weighted by the actual vote returns (both at the precinct level and then overall) the results are certainly more accurate than those leaked on Election Day, and arguably more accurate than most political surveys. They are an unmatched resource, and I remain an exit poll fan despite the apparent glitch on Tuesday.
However, the fact that well educated and well-informed consumers of exit polls are asking this question should be a grave concern to those of us who value survey research. By ignoring the reality that partial exit poll would appear on the Internet, and doing nothing to help consumers data put that data into some context, we have seriously undermined public confidence in exit polls and surveys generally. We can do better.
Related Entries - Exit Polls
The NEP can only authoritatively comment on its own methods, procedures and outcomes, not on those of the Election Voting Machines.
Without a sufficient data set of reliably accurate vote information with which to compare, NEP leaders have absolutely no statistical basis to suggest that the problem lies with the exit polling and not the voting machines.
Basically NEP has simply accepted on faith that the machines numbers are 100% accurate, which we all know is not the case.
Posted by: Night Owl | Nov 5, 2004 10:08:20 PM
Why is there only ONE exit poll taken? Why not have competing exit polls, as there are competing pre-election polls? Is it that expensive that multiple polls are prohibitive? That strikes me as unlikely. We need some healthy glasnost here.
Posted by: vlad | Nov 5, 2004 10:09:50 PM
We all believe that normalized, cross tabulated exit polls give valuable and believable voter demographics and other data. I don't think you need to worry about this so much. :) However, could you give us more background on Final Exit Poll methodologies?
Your readers have suggested potential issues with Final Exit Polls that may or may not potentially introduce bias. A somewhat complete list:
A. Final Exit Polls missed GOP turnout surge somehow.
B. Final Exit Poll gender imbalance.
C. Vote spoilage.
D. Provisional votes. Were respondents asked if they voted provisionally?
E. Shy Republicans and pollsters not allowed close to polls. These were addressed in the article. Are these traditional problems or unique to this year?
It seems that Exit Pollsters must think about these issues as they prepare their final exit poll stategies. Could we learn more about how A-D are dealt with in Exit Poll methodologies?
I hope it is not too much to ask for your further thoughts on Exit Poll methodologies. I would hate to rely on one NY Times article as the best resource on these issues.
Posted by: Alex in Los Angeles | Nov 6, 2004 12:04:17 AM
From the revisions of the exit poll numbers, it looks as if they deleted four points of Latino Kerry voters to move the overall results into line with the actual vote. At the same time, the Latino Bush voters would seem not to have been reduced at all in the revisions, and such that their percentage rose to an unlikely 40+ level. Can it be explained how such results of the exit-poll revisions could have occurred?
Posted by: John S Bolton | Nov 6, 2004 4:22:56 AM
Until voter fraud can be completely ruled out, this all seems like smoke and mirrors. None of the NEP explanations sound reasonable enough to a) say why this election was any different from previous ones and b) explain away enthusiastic Bush supporters (you know, all of Karl Rove's evangelicals?)
Also, do you really think NEP would dare mention voter fraud unless they were 100% sure? Such a claim would kill them, while these low-grade excuses are much easier to hide behind.
I am thoroughly unconvinced.
Posted by: Eric Kreidler | Nov 6, 2004 3:08:31 PM
until someone can explain the data reversals found, for example in the Fl. there will be no rest for the weary. Something is NOT right. The numbers are being manipulated.
Posted by: billjpa | Nov 6, 2004 10:35:19 PM
Hi John Bolton,
I just want to highlight the point you've made.
**The latino vote was very different in the Final Exit Poll.
This is amazing because you have data on a specific discrepancy between the Final Exit Polls and the cross tab exit polls.
1. Do you have more data to post?
2. I guess I was wrong. Obviously, a final exit poll discrepancy with the vote count could call into question the cross tabbed, normalized exit polls, not just the vote count.
Posted by: Alex in Los Angeles | Nov 6, 2004 11:48:06 PM
The above is a deduction based on Dick Morris' report of an unadjusted exit-poll total of 12% for Latino voters, which got adjusted to 8%. This is the bulk of the adjustment made to bring Kerry down, and it causes an anomalous 44% Latino republican vote to be estimated. I would guess that the exit poll staffing was full of people from ethnic studies programs, who wanted to magnify the influence of their groups, and others very much in sympathy with such a project. Then someone upstairs adjusted their numbers, but can't release the data on how they did this; because it was unbalanced in the furthest degree.
Posted by: John S Bolton | Nov 7, 2004 2:24:19 AM
Poll skew? Rove told me all about it. It was all about the questions.
Posted by: Barney Bush | Nov 7, 2004 9:30:24 AM
OK, I'm naive, so I just have a couple of basic questions here:
1. There have been reports floating around that the kinds of exit poll vs. election discrepancies we've seen are a huge red flag (for fraud) when they are seen elsewhere. Anybody know if that's true? And if it's true, are we talking about discrepancies from the early ("Final exit poll") data or the later, revised/normalized/cross-tabulated version?
2. What is the smallest geographic region for which exit poll numbers are available--precinct? county? or are we stuck with state?
Posted by: Abe Fisher | Nov 8, 2004 8:32:25 AM
Fascinating. I hope Mark Blumenthal will comment on your numbers.
1. Dick Morris makes the claim exit polls are used to verify votes counts in countries such as Mexico.
2. Data on Final Exit Polls, used on election night, is hard to locate. There is no systematic source or analysis of Final Exit Polls that Mark Blumenthal has mentioned. There are bits and pieces at various sites. Places like CNN are now reporting normalized exit polls, which are designed to match reported vote counts.
In other words, the American public is not engaged in an informed debate yet on the Final Exit Polls. It is unnecessary and sad.
Posted by: Alex in Los Angeles | Nov 8, 2004 5:15:02 PM
What are the rules and methods used to "revise" exit polls to bring them into correspondence with the tallied vote. Does anyone know?
Posted by: Joel Meyers | Nov 13, 2004 3:19:41 AM
How can a pollster "know" that his exit poll is skewed before any tallies are available? How could NEP rule out the possibility that Kerry was simply doing better than they expected?
Posted by: John Kesich | Nov 17, 2004 3:27:40 PM
The comments to this entry are closed.