A bit of breaking news: The full 77-page "evaluation" of the exit polls prepared by the firms that conducted the exit polls was released this morning and can be downloaded from their website, exit-poll.net .
Some highlights from the Executive Summary:
Our investigation of the differences between the exit poll estimates and the actual vote count point to one primary reason: in a number of precincts a higher than average Within Precinct Error most likely due to Kerry voters participating in the exit polls at a higher rate than Bush voters…
Exit polls do not support the allegations of fraud due to rigging of voting equipment. Our analysis of the difference between the vote count and the exit poll at each polling location in our sample has found no systematic differences for precincts using touch screen and optical scan voting equipment….
Our detailed analysis by polling location and by interviewer has identified several factors that may have contributed to the size of the Within Precinct Error that led to the inaccuracies in the exit poll estimates. Some of these factors are within our control while others are not.
It is difficult to pinpoint precisely the reasons that, in general, Kerry voters were more likely to participate in the exit polls than Bush voters. There were certainly motivational factors that are impossible to quantify, but which led to Kerry voters being less likely than Bush voters to refuse to take the survey. In addition there are interactions between respondents and interviewers that can contribute to differential non-response rates. We can identify some factors that appear to have contributed, even in a small way, to the discrepancy. These include:
- Distance restrictions imposed upon our interviewers by election officials at the state and local level
- Weather conditions which lowered completion rates at certain polling locations
- Multiple precincts voting at the same location as the precinct in our sample
- Polling locations with a large number of total voters where a smaller portion of voters was selected to be asked to fill out questionnaires
- Interviewer characteristics such as age, which were more often related to precinct error this year than in past elections
Seperately, Warren Mitofsky informed the email listserv of the American Association for Public Opinion Research (AAPOR) that the individual level respondent data are being sent today to the Roper Center Archives at the University of Connecticut and the Inter-University Consortium for Political and Social Research (ICPSR) at the University of Michigan. The data at the Roper Center should be available in "about two weeks."
Obviously, I want to read the full report carefully…more later…
The report states that the exit poll results were off by at least one standard error in 30 states! Furthermore, although the average error was 5.5% in Kerry’s favor, the state-by-state error varied greatly, with 26 states being outside one standard error for Kerry and 4 for Bush.
Given this, is there any reason why we should not conclude that the entire poll results are garbage?
Also, any idea why this year’s exit poll error is the worst “that we have data for”? I recall procedural changes were put in place in order to improve the quality of the data.
having read the entire report:
things of special note:
1. interviewers in or post-college had a higher wpe (essentially, error in the direction of one candidate or the other) than others.
2. most interviewers were female
3. primary sources of interviewers included college professor recommendations and craigslist ads
hypothesis: voters (almost certainly accurately) concluded that interviewers were liberal and thus Kerry voters were more likely to talk to them…throw in any sort of additional colloquy engaged in between the interviewers and interviewees and there you have it.
Convincing Evidence That the Exit Poll Discrepancies Were Caused by Bush Supporters Being Less Willing Than Kerry Supporters to Fill Out Exit Poll Questionnaires?
Edison/Mitofsky today publicly released a 77 page report on what went wrong in their Election 2004 exit polls.
Hello Mark, I hope all is going well with you.
I posted as I read the report.
http://dalythoughts.com/index.php?p=2674
One thing I have concluded– I do not buy their hypothesis that Kerry voters were more likely to respond than Bush voters. As I point out in Update five to my post, there is a single table they provide which to me is sufficient in disproving that theory (although there were other places where the data seems to me to not support their supposition as well). On page 37 of the report, they have a chart of completion rate, refusal rate and miss rate by precinct partisanship. The table shows that the completion rate was actually higher in high Republican (Kerry gets less than 20%) and moderate Republican (Kerry gets less than 40%) precincts than in high Democrat and moderate Democrat precincts.
Another example is on page 39 where they note that the higher the population density, the greater the error in Kerry’s favor.
Report Released
…on the election day exit poll travesty. Via FoxNews:Edison Media Research and Mitofsky International found that the Democratic challenger’s supporters were more likely than President Bush’s supporters to participate in exit polls interviews. They al…
Gerry:
huh? that Bush voters would be less reticent to speak to interviewers in heavily Republican areas is hardly surprising (interviewers were local…in other words they were less likely to be young idealistic Kerry supporters in Montgomery Alabama).
as for population density….exactly, urban areas (by definition pro-Kerry) are exactly the areas where Bush voters would be the most reticent to be polled.
I recall from MP’s pre-election post describing exit polling that, in the past, exit pollsters were supposed to approach every nth voter leaving the polling place. Say, every 10th.
But, reading this report one gets the impression that in 2004 the pollsters may have had some leeway in selecting whom they approached. If so, that could easily account for errors of this magnitude.
further thought:
Dean voters were oversampled in the primaries…..tying right into the overenthusiastic liberal college kid hypothesis.
Nathan:
“that Bush voters would be less reticent to speak to interviewers in heavily Republican areas is hardly surprising”
Nathan, you are not understanding what I was getting at.
Let’s suppose that Kerry voters were 40% likely to participate. Let’s say that Bush voters were 35% likely (these are made up numbers to make the math easier).
Now let’s imagine three precincts. Each precinct had exactly 1000 voters. Precinct A was inner city, and 800 people voted for Kerry, 200 for Bush. Precinct B was in Utah. 800 people voted for Bush, 200 for Kerry. Precinct C was in an Iowa suburb. 500 voted for Kerry, 500 voted for Bush.
In these examples, the participation rate for precinct A would be 39%. The participation rate for precinct B would be 36%. The participation rate for precinct C would be 37.5% In other words, the participation rate would differ.
If you are suggesting that in Bush precincts, the Bush supporters would not be as shy, you would have to believe that it would be in an exact proportion so as to make it so that the overall participation rate did not change. That defies credulity (and one would expect to see the other side of the coin, namely Kerryites being shy in overwhelmingly Republican precincts; we also do not see that).
Nathan,
“Dean voters were oversampled in the primaries…..tying right into the overenthusiastic liberal college kid hypothesis.”
Exactly– a point I had made on my blog as well. Your note about the recruiting from Craigslist and by college professors also fits.
Gerry’s blog is a good read.
The evidence leans towards a bias in selection of participants on the part of the pollsters. However, as Gerry notes, that doesn’t explain why precincts in which 100% of voters were selected still showed a significant, if lower, error rate.
There is no evidence confirming the suspicion that Bush voters opted out of the poll or intentionally changed their vote. If either of these were true the error rate would have been more uniform. I think that most people who hold to these theories don’t understand that the exit poll is a secret ballot that is folded up and deposited anonymously.
Digging deeper into the report I find that the “Republicans opted out” theory is even more unlikely. The exit polls and “cross surveys” indicate a 37-37 split in party ID, which is as high as it has ever been for Republicans, and higher than most pre-election polls (except for Gallup).
Similarly, the “Republicans lied to the pollsters” theory also takes a blow. The exit poll found that only 6% of Republicans indicated they voted for Kerry, which is consistent with pre-election polls (actually lower than most).
I’d also like to expand on Nathan’s comment about the primary sources of interviewers being Craigslist and college professors. I agree there is likely some bias introduced in interviewer selection, but the degree may not have been as large as his comment implies. Page 49 lists the primary sources of interviewers in descending order of frequency, and Craigslist is second from last on the list. Previous interviewers, and their recommendations, were the leading sources (not surprising), and local career centers were more common than Craigslist.
Buried deep within–says it all:
“Early in the afternoon on November 2nd, preliminary weightings for the national exit poll overstated the proportion of women in the electorate. This problem was caused by a
programming error involving the gender composition that was being used for the
absentee/early voter portion of the national exit poll. This error was discovered after the first two sets of weighting; subsequent weightings were corrected. This adjustment was made before NEP members and subscribers used exit poll results on-air or in print.”
Mistabulated, highly biased garbage is still garbage. And we still have little proof the exit polls were actually taken, beyond token hirings.
So when did the code change, exactly, relative to the Scoop PDFs? Is that why more data changed, in the early tabulation, going from the restrictive “House” filter to the laxer “President” one, than there were new records added? Maybe some rolled up data was there all evening.
In any event, we can prove a data subset of 2000 records that was always there and never changed, even into the wee hours. So what, exactly, was “fixed” by the code change? We still need raw data, of course, to understand the tabulations.
1. The difference between Pop(3) and Number of Respondents in the Scoop PDFs is exactly 2000 for all sets of “President” files, and regional breakdowns of this subset are all divisible by 4 in all files, implying they are the allegedly corrected data set. [Occam’s razor: there can’t be two data subsets of exactly 2000 records with the 4-fold signature of telephone/absentee data]
2. This subset, and its regional distribution, never changed over the course of the evening, so the program correction, if it affected the regional totals, had to happen before 3.29pm on 11/2.
3. If “President” and “House” tabulations are filters of the same database, then the extra “Presidential” records have more changes to the Pop(3) field than allowed by the difference between Number of Respondents, and the tabulated number of persons answering both “House” and “President” (intersection subset).
4. Therefore, either data were not actually modified by a code fix, as claimed, or the correction was applied early but to only one of the two tabulations.
This leaves the obvious hypothesis: that we can see the signature of the tabulation errors in the PDFs, since the fix was differentially applied. But which tabulation is “correct”???
Folks, if Gerry successfully shoots down Mitofsky’s claim that the early exit polls were wrong because Bush voters didn’t participate, then what are we left with as to why the early exit polls were garbage?
Perhaps what we’re left with, Steve, is that the early exit polls were correct.
Early exit polls have never been correct. All the data shows that they got tighter as the night went on. Unless you’ve seen something I haven’t.
Rick,
Not to be argumentative — my question is a sincere one — which “data…as the night went on” are you referring to?
The last “unadjusted” data I saw, from around 12:30 A.M., showed Kerry winning Ohio and the nation. The next bit of data released by Mitofksy/Edison Media already had filtered in “actual” vote tallies…
As I understand it, *all* the exit polls on Election Day which were not “adjusted” showed Kerry winning Ohio and the nation, and regardless of whether or not the first of the three N.E.P. exit-poll samplings showed a “wider” race than the second and third, can the third really be termed inaccurate, and/or portrayed as just (to paraphrase what I think you’re implying) a “snapshot of a race” which, in some untabulated, off-screen moment, inexplicaby got closer and suddenly switched to Bush ?
Anyway, just looking for a clarification of what you’re saying.
The News Editor
The Nashua Advocate
http://www.nashuaadvocate.blogspot.com/
Gerry,
The completion rates are essentially the same regardless of the partisanship of the precincts, but, in any case, it tells you nothing about the people who did not respond; thus it cannot be used to “disprove” differential response rates. Most of the other data most definitely supports the hypothesis of higher refusal/noncooperation from Bush’s supporters, but it cannot prove it because you still don’t know which candidate the refusers voted for.
The Nashua Advocate, I was referring to american woman’s comment: “Perhaps what we’re left with, Steve, is that the early exit polls were correct.”
An analysis of historical exit polls indicates that as the night goes on and the samples are completed, they have always got closer to the election result (see Table 6.3 of Mitofsky and Edelman 1995 for one example).
Specifically for 1992, “As can be seen [in Table 6.3], the estimates gradually improved as the data became more complete. This should surprise no one. (What surprised the authors were the reporters who thought VRS should have had the result to the last decimal point early in the evening, before the exit poll was complete.)” (Mitofsky and Edelman, 1995, pp. 92)
If the “early” exit polls were correct for 2004, then I’m saying that would be a break from what we know is to be precedent.
Nashua Advocate wrote:
“can the third really be termed inaccurate, and/or portrayed as just (to paraphrase what I think you’re implying) a “snapshot of a race” which, in some untabulated, off-screen moment, inexplicaby got closer and suddenly switched to Bush?”
It is not clear to me how you form this argument? Do you believe that the exit polls switched to Bush somehow at some point?
Trying to keep up with the arguments but this sounds inaccurate.
Thanks!