Those Leaked PDFs

Exit Polls Legacy blog posts

About those newly leaked exit poll numbers. Early this morning, Mickey Kaus linked to my earlier post the recently leaked exit poll documents and declared them a "smoking gun." When I first scanned the documents yesterday, I did not see the same bombshell. I still don’t but, let’s take a closer look.

First, let me clarify the source: Scoop, a left-of-center website in New Zealand that has been following the various vote fraud conspiracy theories, somehow obtained copies of the pdf reports that NEP provided for its national and regional exit polls. Since Scoop provided no state level data, it is likely that their source was a newspaper subscriber, not one of the NEP network partners. By the way, the "scoop" on the scoop numbers came from on New Year’s eve from the Blue Lemur, another "progressive" site.

The documents do tend to confirm what had been reported elsewhere (links to documents are in the bullet points that follow; all times are presumed to be EST):

  • On Election Day, the "national" exit poll had Kerry ahead by three points (51% to 48%) at 3:59 PM and by the same margin (51% to 48%) at 7:33 PM when the polls were closing on the east coast. By 1:33 PM the following day, the completed, weighted-to-match-the-vote exit poll showed Bush leading (51% to 48%). These numbers had been previously reported by the Washington Post‘s Richard Morin and Steve Coll on November 3.
  • The early samples included too many women: The percentage female fell from 58% at 3:59 PM to 54% at 7:33 PM, but this change alone did not alter the overall candidate standings (as the Simon/Baiman paper argues). By the next day, the sample was still 54% female, but the results among men and women were very different – Bush was 4 percentage points higher among men, 3 points higher among women.
  • All of the three releases are marked as "weighted," but keep in mind: The first two releases were weighted only to bring their geographical distribution into line with hard counts of actual turnout. The last release would have been weighted so that it matched the official count (something I explained here).
  • Keep in mind that the 7:33 PM sample from election night was incomplete. It had 11,027 interviews, but the next day NEP reported 13,660. The missing 2,633 interviews, presumably coming mostly from states in the Midwest and West, amounted to 19% of the complete sample (The Simon/Baiman paper includes what appears to be a later and more complete national survey report – more on that tomorrow).

The margin of error provided by NEP for the national exit poll was +/-1%. Thus, Kerry’s early lead and the overall differences between the 7:33 p.m. and 11/3 numbers were statistically significant. That the errors in the national poll were statistically significant, while similarly sized errors in state exit polls were not, owes to the much larger sample size of the national survey.

So is there anything truly new in these documents?

Perhaps not. At least not to me, and hopefully not to MP’s faithful daily readers. However, to political sophisticates like Kaus who do not share our odd obsession er…enthusiasm for the exit poll controversy, the official documents have more power than a few lines buried in an online chat. The last time most political junkies checked in on this story, it was a few days after the election and the "blame the bloggers" meme was in full force. You remember: The problem was not the exit polls or the way they were handled by the networks, but the foolish bloggers who blabbed about "early numbers" they did not understand. Well, these documents confirm something loyal MP readers have long known – the just before poll-closing numbers had the same errors.

Though my perspective is different, I am struck by how little guidance these cross-tabs provide about statistical significance. A newspaper subscriber like Richard Morin would have to do what I did above: Look up the sampling error provided by NEP in a separate table and apply it separately to each number. I cannot imagine that many editors or political writers went to that much trouble.

By comparison, I am told that NEP provided the network "decision desks" with printouts (or computer screens) that provided the exact confidence level for every vote estimate. That is, the estimate for each state included a percentage indicating the statistical certainty of the leading candidate winning the state. Networks would consider calling a state for a candidate only when that percentage went over 99.5% certainty. I had assumed the national poll reports provided to newspapers included similar reports of statistical significance. That they did not may explain why the newspapers that subscribed to the exit polls have been more willing to complain about the exit polls in public.

The continuing stonewall of secrecy that the networks have erected around the exit polls does not help. It is that secrecy, as much as anything else, that continues to fuel the more bizarre conspiracy theories floating around the blogosphere. I remain a skeptic of widespread fraud, but I cannot understand the continuing secrecy: Why did these documents have to be leaked by a left-wing web site in New Zealand? Why did NEP not release them in early November? Why did it take until late December for NEP to make the basic methodology statements the networks had on Election Day available online? And why so much reluctance to discuss, openly, what went wrong and why?

A bit more transparency from news organizations that trumpet our "right to know" would certainly help.

11/4 – Omitted formatting restored

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.