« More on (Sigh...) Exit Polls | Main | The "Smoking Gun" Part II »

January 04, 2005

Those Leaked PDFs

About those newly leaked exit poll numbers. Early this morning, Mickey Kaus linked to my earlier post the recently leaked exit poll documents and declared them a "smoking gun." When I first scanned the documents yesterday, I did not see the same bombshell. I still don't but, let's take a closer look.

First, let me clarify the source: Scoop, a left-of-center website in New Zealand that has been following the various vote fraud conspiracy theories, somehow obtained copies of the pdf reports that NEP provided for its national and regional exit polls. Since Scoop provided no state level data, it is likely that their source was a newspaper subscriber, not one of the NEP network partners. By the way, the "scoop" on the scoop numbers came from on New Year's eve from the Blue Lemur, another "progressive" site.

The documents do tend to confirm what had been reported elsewhere (links to documents are in the bullet points that follow; all times are presumed to be EST):

  • On Election Day, the "national" exit poll had Kerry ahead by three points (51% to 48%) at 3:59 PM and by the same margin (51% to 48%) at 7:33 PM when the polls were closing on the east coast. By 1:33 PM the following day, the completed, weighted-to-match-the-vote exit poll showed Bush leading (51% to 48%). These numbers had been previously reported by the Washington Post's Richard Morin and Steve Coll on November 3.

  • The early samples included too many women: The percentage female fell from 58% at 3:59 PM to 54% at 7:33 PM, but this change alone did not alter the overall candidate standings (as the Simon/Baiman paper argues). By the next day, the sample was still 54% female, but the results among men and women were very different - Bush was 4 percentage points higher among men, 3 points higher among women.

  • All of the three releases are marked as "weighted," but keep in mind: The first two releases were weighted only to bring their geographical distribution into line with hard counts of actual turnout. The last release would have been weighted so that it matched the official count (something I explained here).

  • Keep in mind that the 7:33 PM sample from election night was incomplete. It had 11,027 interviews, but the next day NEP reported 13,660. The missing 2,633 interviews, presumably coming mostly from states in the Midwest and West, amounted to 19% of the complete sample (The Simon/Baiman paper includes what appears to be a later and more complete national survey report - more on that tomorrow).

The margin of error provided by NEP for the national exit poll was +/-1%. Thus, Kerry's early lead and the overall differences between the 7:33 p.m. and 11/3 numbers were statistically significant. That the errors in the national poll were statistically significant, while similarly sized errors in state exit polls were not, owes to the much larger sample size of the national survey.

So is there anything truly new in these documents?

Perhaps not. At least not to me, and hopefully not to MP's faithful daily readers. However, to political sophisticates like Kaus who do not share our odd obsession er...enthusiasm for the exit poll controversy, the official documents have more power than a few lines buried in an online chat. The last time most political junkies checked in on this story, it was a few days after the election and the "blame the bloggers" meme was in full force. You remember: The problem was not the exit polls or the way they were handled by the networks, but the foolish bloggers who blabbed about "early numbers" they did not understand. Well, these documents confirm something loyal MP readers have long known - the just before poll-closing numbers had the same errors.

Though my perspective is different, I am struck by how little guidance these cross-tabs provide about statistical significance. A newspaper subscriber like Richard Morin would have to do what I did above: Look up the sampling error provided by NEP in a separate table and apply it separately to each number. I cannot imagine that many editors or political writers went to that much trouble.

By comparison, I am told that NEP provided the network "decision desks" with printouts (or computer screens) that provided the exact confidence level for every vote estimate. That is, the estimate for each state included a percentage indicating the statistical certainty of the leading candidate winning the state. Networks would consider calling a state for a candidate only when that percentage went over 99.5% certainty. I had assumed the national poll reports provided to newspapers included similar reports of statistical significance. That they did not may explain why the newspapers that subscribed to the exit polls have been more willing to complain about the exit polls in public.

The continuing stonewall of secrecy that the networks have erected around the exit polls does not help. It is that secrecy, as much as anything else, that continues to fuel the more bizarre conspiracy theories floating around the blogosphere. I remain a skeptic of widespread fraud, but I cannot understand the continuing secrecy: Why did these documents have to be leaked by a left-wing web site in New Zealand? Why did NEP not release them in early November? Why did it take until late December for NEP to make the basic methodology statements the networks had on Election Day available online? And why so much reluctance to discuss, openly, what went wrong and why?

A bit more transparency from news organizations that trumpet our "right to know" would certainly help.

11/4 - Omitted formatting restored

Related Entries - Exit Polls

Posted by Mark Blumenthal on January 4, 2005 at 04:38 PM in Exit Polls | Permalink

Comments

Mark, are you aware of your own biases? They're show rather clearly in places like this:

"the completed, weighted-to-match-the-vote exit poll"

They weren't 'weighted to match the vote', they were *adulterated* to match the *claimed tallies*.

Absent empirical validation, the tallies represent only themselves. I really don't think you can justify as good science your claim that they also represent 'the vote'. Whether they represent the vote is the *issue*, and that's not at all an unimportant point.

Posted by: Mairead | Jan 4, 2005 5:46:57 PM

Mark, are you aware of your own biases? They're show rather clearly in places like this:

"the completed, weighted-to-match-the-vote exit poll"

They weren't 'weighted to match the vote', they were *adulterated* to match the *claimed tallies*.

Absent empirical validation, the tallies represent only themselves. I really don't think you can justify as good science your claim that they also represent 'the vote'. Whether they represent the vote is the *issue*, and that's not at all an unimportant point.

Posted by: Mairead | Jan 4, 2005 5:46:57 PM

As always Mark, you have the scoop. I did not know that there was a difference in the tabs sent to the newspapers and the network subscribers. Useful info!

I do find it odd though that the national polls showed Kerry up 51-48 at 3:59, 7:33, and as confirmed by Morin to me, the final round (not sure if this was actually the 7:33 round though).

As indicated in Mitofsky and Edeleman (1995), for the 1992 elections, each round after 8pm showed a tighter margin (but still with a 1.6% bias toward Clinton). The text doesn't seem to indicate that any of these rounds were weighted to the election tallies. It seems to attribute the discrepancy from successive earlier rounds to an incomplete sample.

One question... We know that these data are based on a sub-sample of the precinct samples. We also know that when these updates were phoned in, a count of the non-responses were phoned in. Do you think that the age, race, gender of the non-responses were also phoned in? Also, how do you think they accounted for the non-responses in their sub-sample? Did the sub-sample include the non-responses (e.g., if they took every nth survey, were blank surveys included so that it was equally likely that the non-responses were included in this sub-sample?)

Also, consider that the 2004 primary exit poll methods, conducted by Edison/Mitofsky, included the following methodology statement (as available via the Roper Center),

“The exit poll results are weighted to reflect the complexity of the sampling design. That is, the weighting takes into account the different probabilities of selecting a precinct and of selecting a voter within each precinct. For example, minority precincts that were selected at a higher rate receive a smaller weight than other precincts of the same size. Except for Iowa, there is also an adjustment for voters who were missed or refused to be interviewed, which is based on their observed age, race and sex.”

Do you think this "adjustment for voters who were missed or refused to be interviewed" was included in the leaked data (if of course this is how the survey was conducted this time)? Finally, if our "profiling" of voter intent of those missed or refused is incorrect, what does this say about potential for bias in an exit poll?

Many questions. Most simple methods questions. Why their secrecy? Getting information from these folks is like pulling teeth! But with all the poor statistical analysis of the flawed data currently in the public view, can you understand their reticence?

Posted by: Rick Brady | Jan 4, 2005 6:19:35 PM

You've done some great work explaining the weaknesses of exit polls. Thanks. Two questions: One, since exit polls are essentially cluster samples at a precinct level, why couldn't the raw data be used to point to potential fraud within a specific precinct when compared with tabluated results for that precinct? Two, do polls have any evidentiary weight in a court of law?

Posted by: Blue22 | Jan 4, 2005 6:39:45 PM

So is there going to be pressure on the networks to improve exit polling in 2006? so they actually get it right next time?

Posted by: Brian Dudley | Jan 4, 2005 7:20:09 PM

Warren Mitofsky has a history of opposing frank discussions of poll errors. I encountered his antipathy when I sent Everett Carl Ladd a draft of a meta-analysis of the 1996 presidential polls. I did so because Ladd had commented (in the Wall Street Journal) on the unidirectional character of the errors in those 1996 polls. I then subjected the data that Ladd had tabulated to what may have been the first meta-analysis of poll errors. It showed that the odds were 4,900 to 1 against the notion that these errors were the result of chance (see http://www.psych.purdue.edu/~codelab/PollOdds.html ).
Ladd was interested in publishing this contribution and so he circulated my draft among other members of the polling community. It was mentioned by John Fund in the Wall Street Journal and by Richard Morin in the Washington Post National Edition.
Mitofsky reacted with a campaign of intense vituperation and Ladd hesitated to proceed further. But, coming at the dawn of the web revolution, a revision has remained continuously available at the above listed URL and it has been regularly consulted ever since. And no subsequent election has exhibited egregious polling errors comparable to those of 1996.

Posted by: Gerald S. Wasserman | Jan 4, 2005 7:45:59 PM

At least one exit polling mystery solved. Why didn't Mitofsky release the data? Go over and read Kaus. The findings have to be bad so Mitofsky has to be at fault. There can be no doubt so the poor son of a bitch is going to be flayed alive. Kaus (I'm sure on speed dial from John Ellis) has already started the process.

Mitofsky before release when he was saying that he knew there was a problem by mid-afternoon:

What a professional! Nobody better than Mitofsky! We have to trust Mitofsky!

Once we find out he didn't realize an error by noon

Mitofsky is an idiot! Mitosky knows nothing about polling! Mitofsky is a cheap skate!

It sort of makes you sick. I hope MP sticks with his original feelings about Mitofsky.

One more thing. Stop saying conspiracy theories. That is what, we, as humans do, conspire with each other. That is our great talent as a species. If it weren't for that Neandrathals would be in charge of the world (wait a minute - okay, strike that).

Posted by: Wilbur | Jan 4, 2005 8:03:16 PM

Gerald S. Wasserman, thanks for the source!

Since you say "And no subsequent election has exhibited egregious polling errors comparable to those of 1996" may I presume you have references to similar analysis of the 2000 and 2004 primary data? (what about other exit polls like the LA Times exit poll?)

It seems to me that even though exit poll data is available via the Roper Center, there would be lots of statistics grad students looking for a dissertation topic, eager to dive into it. There is really very little published on the subject, which I still find to be amazing considering the academic mantra - "publish or perrish."

Blue22, I don't know where, but I think MP has answered part of your first question (indirectly) about the individual precincts.

The polls are taken so that about 100 samples will be drawn throughout the day (set in advance, they try to approach every nth voter, but miss some and others refuse). The data leaked and linked to by MP, Kaus, and others includes a subsample of the precinct samples. (i.e., they only report about 50 of the 100 on election day). This seems to me to add error as it is a random sample of a random sample of a cluster sample (hence the design effect). Therefore, analysis of the full survey count per precinct should be more "accurate," but remember, this is only 100 samples. An independent sample size of 100 assuming simple random sample has a margin of error of ~+/-10%. Also remember that not every precinct was sampled and therefore if you suspected a certain precinct of fraud, it may or may not have been in the sample.

Suppose though that someone identified a precinct in advance that could have fraud and decided to man that precinct with enough pollsters to approach a little more than 1000 voters, then you could get a MoE of about +/-3% for that precinct. Assuming no other non-sampling error, is +/-3% good enough to stand up in court?

Maybe a lawyer could answer this. But I don't think we need a lawyer to tell us that +/-10% wouldn't stand a chance.

Posted by: Rick Brady | Jan 4, 2005 8:48:09 PM

Rick Brady, thank you for the chance to clarify my remark. I should have said: "And no subsequent 'national' election has exhibited egregious polling errors comparable to those of 1996". To be comparable, a set of national polls would have to be of comparable size; the work of 8 major polling organizations had been considered by Ladd in 1996. And their 8 errors would have to almost all go in one direction. And some of the errors would have to have been extremely large, near 10%. That has not happened since 1996; if it had, the spreadsheet I created would have allowed anyone to produce another meta-analysis quite readily.

Posted by: Gerald S. Wasserman | Jan 4, 2005 9:26:29 PM

Gerald, first question. Which Ladd 1996?

Ladd, E. S. 1996. The election polls: An American Waterloo. Chronicle of Higher Education, 22 November, p. A52.

Ladd, E. S. 1996. The turnout muddle. P.34 in America at the polls: 1996. Storrs, Conn.: Roper Center for Public Opinion Research.

Also, I'm still confused. How do you then justify this statement: "And no subsequent 'national' election has exhibited egregious polling errors comparable to those of 1996" if, as it appears, there has been no analysis of a national election?

Posted by: Rick Brady | Jan 4, 2005 10:31:57 PM

The 2004 election saw many more GOP voters much more disrespectful of mainstream media than the 2000 election. You had the Rather debacle and the NY Times debacle, but also the steady growth of talk radio audiences and cable/Fox News. When they left the polling places and a "media" representative asked them how they voted, why wouldn't they lie to screw him up? It's not like all of them had to lie, but how about 6 in every 100? I was an avid Bush voter who was hoping to be accosted when leaving the polls so I could lie, but I realize I'm not typical... If Mitofsky people know their business, I think this must be the key to your exit poll puzzle.

Posted by: Larry | Jan 5, 2005 12:03:10 AM

MP, though I am not schooled in the intricacies of polling, I find you very informative. Thanks for sharing your knowledge with us in this forum.

The accuracy of the exit polls need not be argued in a vacuum.

The Mitofsky exit polls and the official tally are not the only data points that we have. The New Hampshire hand recount of largely Optiscan ballots was deemed accurate by all involved parties. NH had the largest exit poll discrepancy of any battleground state according to Freeman. Thus, one can conclude that physical ballots were stuffed in New Hamsphire or, more logically, that the exit polls were wrong.

Moving to Utah, the state with the twelfth largest discrepancy according to the Simon/Freeman data, the BYU exit polls agree with the official tally while the Mitofsky exit polls do not. (Kerry numbers: Official Tally 26.0%; Mitofsky 29.1% (both from MIT/Caltech Voting Project); BYU exit poll: 26.5% (from exitpoll.byu.edu). Thus, one can conclude that, in addition to the ballots being stuffed, the BYU exit polls were stuffed too, or one can conclude that the Mitofsky exit poll was wrong.

At this point, it should be patently obvious that the exit polls were wrong. Energy should be focused on analyzing why they are wrong to improve the process in the future rather than using the exit polls to allege fraud.

Posted by: Marty H | Jan 5, 2005 12:24:47 AM

I did some quick math: if the 7:33 p.m. sample was correct and totalled 11,027 voters with a 51-48% Kerry lead, for the final results (with 13,660 voters) to show Bush leading 51-48%, Bush had to have won 64% of the votes in that last portion of the sample. Which on the face of it is monstrously out of line with both the actual election results and the previous exit polling. Would a pollster, the day after, "cook" his numbers so that they mirrored the actual vote percentages? If he wished to be rehired, at $10 million a pop, there's a strong incentive for him to do so.

Posted by: Bruce Allardice | Jan 5, 2005 2:34:10 AM

Hello, Mr. Blumenthal. I wonder why our government and press do not question exit polls in the Ukraine. Is exit polling more advanced there? Thank you.

Posted by: Dwight Van Winkle | Jan 5, 2005 3:05:58 AM

Rick Brady, the information you want was given in: http://www.psych.purdue.edu/~codelab/PollOdds.html

Posted by: Gerald S. Wasserman | Jan 5, 2005 9:01:26 AM

Gerald S. Wasserman, thanks again, but it doesn't answer all of my questions. Especially about how you justify the statement, "And no subsequent 'national' election has exhibited egregious polling errors comparable to those of 1996."

One thing that I didn't find in your paper is the exact date of each of those pre-election polls that you cite. If any of those polls were conducted more than a day or two before the election, I'd say their predictive value is about worthless. Polls are supposed to be a snapshot in time and any number of things could have changed the minds of voters in the last few days before an election.

Also, not in there is information about the assumptions used to assign undecided voters (if they were assigned).

These are important considerations I would like to think.

Posted by: Rick Brady | Jan 5, 2005 10:00:08 AM

I'm really astonished by all the implicit 'nothing to see, move along' that's going on here.

Would someone like to try making a case that it's good science to silently 'correct' the exit polls with numbers from the tally?

How about a case for the accuracy of the tallies themselves?

Who benefits when people are given the 'move along' treatment when the stakes are high and unanswered questions abound? Who benefits from the obfuscation and damping-off of the issues?

Shouldn't we expect impartiality and accurate representation of the facts from those who claim the mantle of science?

Posted by: Mairead | Jan 5, 2005 10:50:26 AM

Mairead, "Would someone like to try making a case that it's good science to silently 'correct' the exit polls with numbers from the tally?"

I guess the case is that: 1) it has always been done (that's what I hear anyway); and 2) it goes to show that the pollsters are (and presumably have been) more confident in the election tally than they are of their polls.

Someone who used to comment on this blog described it like this to me in an e-mail: The election tally is "considered" to be like an atomic clock (infallibly accurate) and one can compare the exit polls to a VERY expensive Rolex Watch.

We notice that the Rolex Watch is way off from the atomic clock. In the past, the makers of the Rolex Watch simply reset it to the atomic clock and blamed the error on external forces that act on the watch and cannot be predicted or accounted for by the watch maker. "After all, the Rolex is no atomic clock" they say.

This time, the difference is rather large; much larger than normal.

Some say that this suggests that someone fudged the atomic clock when no one was looking (Freeman, Simon/Baiman). Others say the Rolex Watch was likely tinkered with to trick the atomic clock (Dick Morris/Jim Geharty). Still others seem to think that the Watch is fine; this time, as times in past, it was subjected to a force outside the control of the watchmaker (Mitofsky/Lenski).

I say, perhaps it is a combination of all three? Who knows?

So now the question is: Do we look at the Rolex Watch to determine that someone tinkered with the atomic clock?

Posted by: Rick Brady | Jan 5, 2005 11:04:28 AM

Mairead, let me correct myself. According to Mitofsky and Edeleman's (1995) review of the 1992 exit polls, the final exit was weighted to the election result. I can only guess that it was like this before and it will be like this in the future.

Posted by: Rick Brady | Jan 5, 2005 11:25:20 AM

Larry,
Its good to lie to the exit pollster, that way no one will believe Bush was elected legitimately, instead they will believe the election was fixed.

Posted by: Brian Dudley | Jan 5, 2005 11:36:32 AM

This is all very interesting, but essentially useless until all of the raw data with the precinct identities is released. My own conjecture is that the nonresponse was asymmetrical by a factor larger than the models (whatever they are) predicted. If the pollsters did their jobs correctly, this should be detectable by the age/gender/race of the non-responders. I suspect that the NEP knows what the problem is, but they may never want to admit that the polls were wrong.

Posted by: Yancey Ward | Jan 5, 2005 11:59:13 AM

Yancey, re: "I suspect that the NEP knows what the problem is, but they may never want to admit that the polls were wrong."

If you read MPs previous posts, you'll find statements attributed to Mitofsky and Lenski that admit that the polls were wrong.

Also, I suspect many of the skeptics (not conspiracy theorists) will not have faith (heh) in the raw precinct data when it is finally at the Roper Center. Mark my words.

Posted by: Rick Brady | Jan 5, 2005 1:11:38 PM

Rick, you seem to persistently respond to me in a way that's not really a response.

I asked whether anyone wanted to try a defence based on scientific principles.

You gave me back one based on bureaucratic/tribal ones ('it's always been done that way'; 'what was good enough for our ancestors is good enough for us').

----------------------------

I'll ask again (of everyone, not of you, Rick): anyone want to try to justify the 'correction' practice based on scientific principles? Does anyone even think it's *possible* to so justify it?

If it's not possible to justify it, what does that suggest about the motives of the 'scientific' pollsters like Mitofsky et al.?

Posted by: Mairead | Jan 5, 2005 4:13:37 PM

Mairead, the scientific case as to why the tallys are more believable than exit polls is simple.

A) widespread fraud with no whistle blowers is hard
to believe.

B) quite frankly the exit polls have such a large non-response that i believe it is unrealistic to expect the exit polls to be correct.

But that leaves a nagging question...why do they have no plans and fixing the non-response problem with the exit polls? this whole thing stinks.

Posted by: Brian Dudley | Jan 5, 2005 5:07:02 PM

Shutting my mouth...I can hear the crickets chirping...

Posted by: Rick Brady | Jan 5, 2005 5:27:29 PM

The comments to this entry are closed.