This site aims to be about political polling and opinion surveys, not about voting irregularities or election reform. Nonetheless, since the controversy over the exit polls has been at the center of the debate over alleged irregularities in the 2004 election, our conversation has sometimes touched on that issue but never addressed it thoroughly. Last night, a prominent FOMP (friend of Mystery Pollster) alerted him to a "preliminary" yet comprehensive report that was issued just before Christmas by the National Research Commission on Elections and Voting. MP recommends it highly to those with a genuine desire to understand what we really know (and don’t know) about the shortcomings in the way we count our vote. Anyone interested in electoral reform should read it in full.
The report has received remarkably little attention both in the mainstream media and the blogosphere (NashuaAdvocate excepted). I say remarkable for two reasons. First, the authors represent an all-star team of non-partisan academic specialists in political science, sociology, statistics and election law. Second, the report is a uniquely fair, rational and exhaustive review of the available evidence on all the major vote count controversies. Those journalists still following Mickey Kaus’ original call for "debunking and more debunking" ought to give it a thorough read. Keith Olberman, this means you!
Here are the key findings from the report’s executive summary:
- Discrepancies between early exit poll results and popular vote tallies in several states may be due to a variety of factors and do not constitute prima facie evidence for fraud in the current election.
- Recent studies noting disparities between county registration rates and voting outcomes in Florida, as well as apparent "machine effects" favoring George W. Bush, are of limited significance and cannot be considered as evidence of election fraud.
- Ohio witnessed significant variability in wait times in some districts, sporadic instances of machine malfunctions, and possible voting tabulation errors, undercounts, and overcounts. Based on data available to this working group, it is extremely unlikely that the absence of these irregularities would have shifted popular vote tallies sufficiently to change the declared winner in Ohio. However, continuing uncertainty over the extent of irregularities merits closer public scrutiny and full disclosure of relevant data.
- A definitive resolution of some allegations of malfeasance or irregularities in the most recent presidential election may never be possible, due to inadequate data and insufficient transparency of the election administration process in many states.
- To restore public credibility in our election system, and to ensure the effective resolution of electoral process controversies in future elections, full and transparent collection and public disclosure of electoral process data are vital.
The section on the exit poll controversy plows ground that should be familiar to regular MP readers. Here’s the money quote:
Although these disparities have alarmed many observers, for several methodological reasons there is no a priori reason to believe that these differences reflect problems with the actual vote tallies. Rather, exit polls as currently designed and administered in the United States are not suitable for use as point estimators for the share of votes that go to different candidates. Their results, in conjunction with other elements of statistical models used by the National Election Pool (NEP) and the decision desks of their news organization members, are best suited for determining the difference between the two leading candidates and whether it is safe to call a particular race for one of them. Furthermore, the current design of exit polls is not well-suited to estimating whether certain aspects of an election functioned properly or not (for instance, efforts to assess whether particular types of voting machines were accurate).
Finally, for those who think that calls for greater transparency come from bloggers, there is this graph:
To ensure that the public and researchers are fully able to assess the significance and limitations of current and future exit polls, this working group recommends that methods, data, and weighting procedures should be fully disclosed for all exit polls in accordance with accepted public opinion survey research practices, such as those endorsed by the American Association of Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP). The timely disclosure of such information would allow public observers to ascertain the significance and limitations of purported findings.
I couldn’t agree more.
One last thought. Like the authors of the report, I have been a skeptic of claims that the exit polls present "a priori" evidence of vote fraud. My single-minded focus on that point represents the narrow focus of this blog. However, conspiracy theories aside, very legitimate concerns remain about the variability in wait times and the distribution of voting equipment, legitimate counting of provisional ballots, use of voting equipment without a paper audit trail and a lack of a transparent process free of partisanship to count and verify the vote. Kos, over at DailyKos, summed it up this way:
The wacked out conspiracy theories hijacked the [electoral reform] issue, taking it away from the obvious travesties — the long lines in Democratic precincts, attempted voter intimidation, etc., to the realm of fantasy… All the crying wolf is hurting the cause for electoral reform.
Kos is right. Electoral reform is important. Those of us who support it need to start talking to the 87% of Americans (and 78% percent of Kerry Voters), according to the Annenberg National Election Study, who are "confident" that this year’s vote was "accurately counted."
Anybody know what the LA times exit poll found for the election? or did it not have a large enough sample to warrant looking at it?
I believe you misread the study, possibly because of your training. What you have to understand is that much of social science now acts from Karl Popper’s model for development of evidence. I would say almost certainly the people working on this report were working from that model (actually those without knowledge of the philosophy of science, which includes the philosophy of social science, probably didn’t realize they were doing that).
Popper works from a sort of nullification theory of evidence. That is you start with a hypothesis and then you attempt to knock it down. The better a theory is able to be disproved the more the theory is brought in to question.
What seems to have happened was that the panel’s starting point was that there was no fraud in the election. They couldn’t start out with the idea that there was fraud because then you would have to prove there wasn’t fraud. Something people just don’t seem to get is that you cannot prove a negative, especially in Popper’s scheme because you just keep throwing instances where there wasn’t fraud and you get nowhere.
Now we start out with the hypothesis there wasn’t fraud. Can we prove that there was fraud. There are two types of proofs we can attempt, one is a priori and one is a posteori. The a priori proof is the stronger of the two but almost impossible. In other words you can’t say because A happened that B is definitely disproved. This would suggest an absolute and irrefutable connection. Nobody but the most radical election theorists was suggesting such a connection.
But, in their wording the authors are suggesting there may be an a posteori proof that there was fraud. That is if you combine the findings with other findings, as a result of further direct study there is a chance that you will find proof (built up over time) that the hypothesis there was no fraud in the election was false.
Actually the panel was sort of saying the opposite of what MP has been saying. MP started with hypothesis that there was fraud and that this hypothesis had been disproved by certain evidences.
So dispense with patting yourself on the back, at least in relation to this study.
“Actually the panel was sort of saying the opposite of what MP has been saying. MP started with hypothesis that there was fraud and that this hypothesis had been disproved by certain evidences.”
I don’t think so Wilbur. Specific instance?
Stats is all about the null hypothesis.
Rick,
There is a difference between tools for the development of a hypothesis and the development of a hypothesis. The second is a far more meta issue. MP follows the hypothesis that there was election fraud because of the type of evidence and analysis he presents and the way he presents it (it has nothing to do with what he really believes). MP presents contrary evidence as, “You see, look at this analysis that shows that there could not have been election fraud.” Almost all of his posts on the exit polls have fallen into this model. He seems to be saying, “All right you guys are saying there is election fraud based on these exit polls but for instance when you look at the methodology…or you see that different things that could have happened it proves that there wasn’t election fraud.”
What the panel was doing was saying there is no election fraud. If you look at the anomolies you can not make an a priori argument that our hypothesis is incorrect – but there is room for further study. As I said I believe at least some of the people on the panel did not understand this is what they were doing.
Your playing some sort of semantics that I either can not follow or choose not to follow. All MP is saying from what I gather is that EXIT Polls are not evidence of fraud, at least not at the level they have been analysed by the general public.
Mark,
Speaking of fraud, this makes me raise a question, have you seen the analysis of the North Carolina election, which claims that people who voted early for non-federal elections…voted the same way as people who voted on Nov 2. But the people who voted early for federal elections differed significantly from people who voted Nov 2.
Brian,
I noticed those NC results as well. I dared not bring them up earlier, because the logic of the analysis was somewhat elusive when I first read it.
The analysis is from Democratic Underground, no? Can you point to an authoritative report?
Brian – I don’t know any more about the LA Times exit poll than what they released online. This PDF file has results but no methodological information:
http://www.latimes.com/media/acrobat/2005-01/14936018.pdf
Much of the same data is reproduced by the PollingReport:
http://www.pollingreport.com/2004.htm#Exit
Rick – I have seen some online speculation about North Carolina but have not seen any comprehensve analysis. That doesn’t mean it’s not out there — I just haven’t seen it.
Oops..that was Brian that asked about NC, not Rick. Apologies. I second Alex’s question — if anyone has a link, please post.
Another oversite: The link I posted was for the LA Times California Exit poll. The PDF for their national survey is at this link:
http://www.latimes.com/news/custom/timespoll/la-110404superchart-g,1,6130757.acrobat?coll=la-news-times_poll
Wilbur,
Please re-read this post:
http://www.mysterypollster.com/main/2004/11/the_freeman_pap_1.html:
“However, as I wrote yesterday – and it bears repeating – THE FACT THAT THE EXIT POLLS SHOW NO EVIDENCE OF VOTE FRAUD DOES NOT DISPROVE VOTE FRAUD. It may have occurred on a scale too small to be detectable by the exit polls.” (emphasis added)
Also note that MP’s post on the NRCEV report does not characterize it in any way! He simply quotes from it. If you want to attack a particular finding of the NRCEV report, then go for it!
I’m noticing a trend here. I’ve posted several times critical of Freeman and Simon/Baiman, only to be grossly mischaracterized as saying that either the exit polls are explainable by random chance or denying the possibility of vote fraud.
I’ve done neither!
Anyone who takes the time to read my posts realize that I’ve said you cannot “prove” vote fraud from the exit poll data in the public realm. The discrepancy could be explained by a combination of vote fraud, sampling error, and non-sampling error.
MP is doing a noble job of being fair in application of his expertise to a very complicated matter.