« Lessons: Mobile Phones | Main | The Freeman Paper »

November 15, 2004

Exit Polls: CalTech/MIT Report

Two new reports on exit polls came to my attention over the weekend. Both reports were written by high-powered PhDs from high-powered institutions, and both contribute to the ongoing debate (or efforts to debunk) theories of vote fraud in this year’s election. Unfortunately, I am not sure either report brings us much closer to resolving the underlying controversy.

I'll take up the first report on “Voting Machines and the Underestimate of the Bush Vote” from the CalTech/MIT Voting Project in this post. The folks at the Voting Project set out to debunk one popular theory initially floated on the web -- that exit polls showed a greater discrepancy with actual results in states with newer electronic voting machines that lack a paper trail.

They knew from their own extensive work on the new voting technology that most states used a mix of voting machines in 2004. So they tried something novel: They produced graphs that plotted the percentage of votes cast by various voting machines (paper ballots, lever machines, optical scan and touchscreen) in each state against the percentage discrepancy on the exit poll for each state. If, for example, touchscreen (or DRE - “direct recording electronic”) voting enabled widespread fraud, then the size of the discrepancy between the exit polls and reality should increase as the percentage of votes cast by DRE increases. The charts show no such pattern by any type of voting equipment, so the CalTech/MIT researchers concluded, “there is no evidence that electronic voting machines were used to steal the 2004 election for George Bush.”

One problem though. As an emailer to RottenDenmark pointed out on Saturday, the CalTech/MIT report used the "corrected" exit poll data now available online at CNN.com. As regular readers of this site know, NEP weights (or adjusts) the exit polls so that their tabulations of vote preference match reality. This is a long-time standard practice for the national network exit polls.

How do we know that the CalTech/MIT report used the corrected data? Remember what you learned in college, “always read the footnotes.” Footnote #2 tells us:

The exit poll data were taken from the cnn.com web site. The poll data can be accessed through http://www.cnn.com/ELECTION/2004/pages/results/index.html. Because the web site does not report the bottom line candidate percentages directly, we had to calculate them from the demographic breakdowns. In this case, we estimated the Bush percentage of votes in the exit polls using the gender breakdown. For instance, 54% of the respondents in Florida were women, 46% men. Women gave 50% of their votes to Bush, men, 53%. Therefore, Bush’s overall share of the exit poll in Florida was calculated as (54% x 50%) + (46% x 53%) = 51.38%.

Now go to the CNN site, and retrieve the results for Florida. You will notice that the current vote-by-gender numbers reported by CNN match the numbers in the above footnote. In other words, the numbers analyzed in the MIT/CalTech paper are the numbers NEP weighted to match the actual result. They are NOT unweighted, end-of-day poll results that have been the object of all the speculation, the ones that showed Kerry doing better in most states than he did in the actual count. Of course, the final weighted vote numbers used in the MIT/CalTech report still show very small, seemly random discrepancies. This is presumably due to rounding of the four percentages (vote by gender and the percentage male and female) they used to calculate each candidate’s vote for each state.

In case there is any doubt, the CalTech MIT report also noted three “outliers” – three states whose exit poll numbers looked very different from the actual result in Footnote #5:

Rhode Island gave 47.4% support to Kerry (sic) in the exit poll, compared to the actual 38.9%. The two other statistically differences were Oklahoma (59.2% exit poll vs. 65.6% official return) and New York (31.9% exit poll vs. 40.5% official return). Note that none of these states was a “battleground state.”

Two problems here. The first is that they obviously meant that Bush, not Kerry, got 38.9% in Rhode Island, 65.6% in Oklahoma and 40.5% in New York.  A simple typo, no big deal.  Now go back to the results at CNN.com and try to replicate their calculations for these three states. The numbers are now different. I get the following percentages for Bush on the exit polls: 39% in Rhode Island , 65% in Oklahoma and 41% in New York. Obviously they now differ from reality by no more than a single percentage point. The CalTech/MIT researchers probably grabbed results for these three states before NEP got around to weighting them to match actual results.

Thus, the charts in the CalTech/MIT report don’t really tell us much. They are essentially analyzing rounding error.

Now, I am assuming this is just an honest mistake. It happens to the best of us.  Moreover, the Voting Project researchers were on the right track.  They can probably replicate this analysis using data captured before NEP weighted the exit polls to match the count.  I would not be surprised if they reached the same conclusion [Clarification: I am guessing the analysis will show no significant relationship between the type of voting equipment and the exit poll discrepancy].

More important, as noted here before, the analysis that the Voting Project researchers attempted could be done with far more precision and power using the raw exit poll data. The exit polls track type of voting equipment down to the precinct level. So if large discrepancies occurred in DRE precincts nationwide, but nowhere else (a very big “if”), the data would show it. Again, I’m dubious, but this would be a very easy theory to “debunk.”  Unfortunately, NEP officials are so far reticent to discuss their data.

So...

Attention Keith Olbermann!: You want to “continue to cover [voting angst] with all prudent speed?” Excellent. Here is one piece of the puzzle you can help solve. The good news is, you don’t need to find some “Deep Throat” informant or submit a Freedom of Information Act request. Just call up NBC’s polling director and ask. OK, true, you may need to convince a few colleagues at the other networks to do the same. Nonetheless, the networks own and control the NEP exit poll data, so I'm sure they’ll gladly help “debunk” this controversy by making the relevant data available. Right? Or does that whole “right to know” thing only apply to everyone else?

Next up, another exit poll report by another MIT PhD...

Related Entries - Exit Polls

Posted by Mark Blumenthal on November 15, 2004 at 06:36 PM in Exit Polls | Permalink

Comments

Thank you! Your final point is spot on, Mark. Free the exit poll data! Keith, if you're reading...tell them to free the data!

Posted by: Sade | Nov 15, 2004 9:30:55 PM

P.S. I've been waiting for this post for days. Thank you!

Posted by: Sade | Nov 15, 2004 9:32:58 PM

Mark,

Thank you for highlighting the errors in the Caltech/MIT report. I will send them a link of your analysis.

This episode highlights the importance of getting the proper data and BACKGROUND knowledge into the hands of institutions willing to perform this "audit" function for the good of our democracy. I hope MIT/Caltech are not discouraged from continuing.

However, even if they and others get the raw data, analysts will need more information on what they are actually analyzing. Namely, how were precints chosen? What weighting is required to make the data "usable?"

***Were precincts chosen randomly, meaning they can treat the sample as representative of the state as is?

***Or were the precints chosen because of the needs for key precints, demographic data requirements, etc, and so the sample is not necessarily representative/random? If so, can the data from the non-random sample be made representative of the state? How?

***Will they need to recreate Mitofsky's non-respondent adjustments based on gender and age?

***Or will the data they receive already contain those adjustments using Mitofsky's expert algorithms?

It seems anyone who will attempt to analyze the raw data of the exit polls will require a primer on exit polling. They will also require specific guidance on the "randomness," quality, and limitations of the samples, which may not be obvious to a non-pollster. And that is why this blog is indispensable reading yet again.

Thank you.

Posted by: Alex in Los Angeles | Nov 15, 2004 9:35:59 PM

Mark, you write:

"Moreover, the Voting Project researchers were on the right track. They can probably replicate this analysis using data captured before NEP weighted the exit polls to match the count. I would not be surprised if they reached the same conclusion."

Which "same conclusion" would you be not surprised that the CalTech/MIT researchers would reach if they use the raw exit poll data? The conclusion the CalTech/MIT report originally reached? Or the theory that "exit polls showed a greater discrepancy with actual results in states with newer electronic voting machines that lack a paper trail"?

Posted by: aaa | Nov 15, 2004 10:10:32 PM

Could you please comment on Dr. Freeman's paper http://www.solarbus.org/stealyourelection/articles/exit-poll-discrepancy-1110.pdf

Also can you explain why exit polls can be used to verify elections elsewhere but in the US they are "known" to be wrong by midday?

Thank you.

Posted by: John Kesich | Nov 16, 2004 4:12:49 AM

They are not likely to find differential results by types of machines used. How can the impossible exit-poll results reported by Steve Sailer in "Bush didn't win 44% of hispanic vote- the smoking exit poll" be explained other than as deliberate faking of data? In dozens of states, the Latino vote proportions of which are not reported separately, the Bush Hispanic support would have had to go impossibly up to and well over 100%, to fit the regional data given. Didn't the exit pollsters start with a model of the non-hispanic white vote six points down from '00, and when this caused exaggeration of the Kerry share, didn't they eliminate Latino Kerry support, to make up the difference?

Posted by: John S Bolton | Nov 16, 2004 5:24:11 AM

aaa asked:

Which "same conclusion" would you be not surprised that the CalTech/MIT researchers would reach if they use the raw exit poll data? The conclusion the CalTech/MIT report originally reached? Or the theory that "exit polls showed a greater discrepancy with actual results in states with newer electronic voting machines that lack a paper trail"?

The former -- see clarification in the main post

Posted by: Mark Blumenthal | Nov 16, 2004 7:17:47 AM

John asked...

"Could you please comment on Dr. Freeman's paper http://www.solarbus.org/stealyourelection/articles/exit-poll-discrepancy-1110.pdf"

That will be next up. Stay tuned...


Posted by: Mark Blumenthal | Nov 16, 2004 7:18:52 AM

Can't wait for the Freeman critique. I have already fed my rational, but unsupportive, comments to him and hope for corroboration from Mark. Freeman says he got over 1000 emails in the three days after this draft paper made it to the internet.

Posted by: esp | Nov 16, 2004 11:55:08 AM

Mark,

Caltech/MIT has made a preliminary look at Dr. Freeman's data. You were right, they did not find indications of machine type bias.

However, they also do not acknowledge their study uses revised exit polls. I'm not sure why they are stubborn on this point.

Here is their email response:
Dear Alex,
Thank you for your questions and comments about our report. We did the
analysis reported there on the data that was publically available to us.
We plan to do an update on the
report, once the final election results are in. At that time, we'll
incorporate the official election returns into the analysis, as well as add
information about what happens when we analyze the data that some on the web
claim were posted on the cnn web site earlier in the evening. Our
preliminary analysis of this other data is consistent with our original
finding that there is no relationship between the type of voting machine
used and the size or direction of the discrepancy.

Posted by: Alex in Los Angeles | Nov 16, 2004 2:09:34 PM

Neater version:
Dear Alex,
Thank you for your questions and comments about our report.

We did the analysis reported there on the data that was publically available to us. We plan to do an update on the report, once the final election results are in. At that time, we'll incorporate the official election returns into the analysis, as well as add information about what happens when we analyze the data that some on the web claim were posted on the cnn web site earlier in the evening.

Our preliminary analysis of this other data is consistent with our original finding that there is no relationship between the type of voting machine used and the size or direction of the discrepancy.

Posted by: Alex in Los Angeles | Nov 16, 2004 2:13:35 PM

"They knew from their own extensive work on the new voting technology that most states used a mix of voting machines in 2004."

The Voting Technology Project folks have one other area of expertise. "Spoiled" votes, those millions of votes that are never counted because of faulty equipment or voter error.

I would love if they would do a similiar graphing of % votes spoiled/provisional by % exit poll discrepancy.

I'm surprised they did not think of doing this as their prior reports strongly highlighted the problem of spoiled votes, and it seems to me that vote spoilage/provisional votes is an obvious source of exit poll discrepancy.

Posted by: Alex in Los Angeles | Nov 16, 2004 6:51:49 PM

I'm new here. I usually post on http://www.majorityreportradio.com

We've found VERY strong links between the CALTECH/MIT Voter Technology Project and a very deep, dark Right Wing -- Republican outfit called the Hoover Institution -- which has very strong links to the Bush administration. We've got some other stuff that pretty much paints them as a spin-machine. Email me if you want more info.

Great news out of Florida tonight!

fishy@tac-wy.com

Posted by: Fishgrease | Nov 16, 2004 10:37:14 PM

I don't think David Baltimore (president of Cal Tech, a scientist, and a Nobel laureate) and Charles Vest (president of MIT), who are leading representatives of CalTech/MIT Voter Technology Project, would be unduly influenced by the Hoover Institute. It is disappointing that they didn't know that polling number they got from CNN was not adjusted to approximately (but not exactly - I think this is a major cause of confusion) match actual results. Then again, I suspect most of us didn't know either; I sure didn't and wouldn't know now if it weren't for this site.

Posted by: Ho-Yon | Nov 17, 2004 2:06:02 PM

I suffered through my fair share of statistics in both undergrad and grad school (MBA), and Dr. Freeman's analysis is both elegant in it simplicity, and yet rigorous enough that I can not find any notable flaws in the methodology.

It's basically a standard statistical analysis of the sample sizes, variances, along with a few assumptions about exit polling and votes that are reasonable, yet the basic probablity analysis produces a compeling result that supports some sort of systemic vote "mistabulation." Indeed, Freeman used the final 6pm exit polling data was weighted by NEP for voter-turnout/gender/subgroups, etc. So, it will interesting to see if the media addresses this study...

Posted by: Observer | Nov 17, 2004 5:47:58 PM

Exactly! Dr. Freeman's and the VTP's reports are at perfect odds. One is wrong. Every time the least bit of evidence surfaces that there might have been fraud this month in Florida, the VTP issues another report. It's what they do. I think it's why they're there. They're where the whole Dixiecrat argument came from. Don't view it if you're looking for a anything scientific.

http://www.vote.caltech.edu/Reports/Florida_discrepancy3.pdf

Back in 2001, right when the Congress was freaking about elections, Stephen Ansolabehere of the VTP and A Fellow of the Hoover Institution,

http://www.pfaw.org/pfaw/general/default.aspx?oid=11948

gave this presentation to the US House of Representatives Science Committee:

http://www.house.gov/science/full/may22/ansol.htm

In that presentation, you'll find plenty about keeping people from voting twice but you'll find absolutely nothing about how we might keep companies like Diebold from doing anything they like with the vote data.

We've alerted several groups to the VTP and they're being looked at by real investigators (I'm not) so there will be more.

Posted by: Fishgrease | Nov 18, 2004 12:54:55 AM

Fishgrease:

The point is they used the wrong data this time. Their methodology is sound. As this post states, given the raw data from the NEP, their methodology would be a contribution to the debate.

Of course, the NEP is doing a much finer grained analysis than the VTP, so the VTP study may be irrelevant once the NEP finishes their report.

Posted by: Alex in Los Angeles | Nov 18, 2004 5:44:19 PM

~~~ The point is they used the wrong data this time. ~~~

No. I don't believe that is the point. Myself and several others are looking at the VTP very closely and it looks like they're right there with several other organizations who set up unauditable electronic voting as something tolerable in this country well before the 2004 election. Did you read my link about the Hoover Institution?

Unauditable election devices are ridiculous. They're a giant step backwards for this democracy and I refuse to accept them as the status quo. VTP is not alone in supporting them... and, in immediately coming out with competing research every time the least bit of evidence lending itself to election fraud in Florida. VTP is just the only one for we've established firm links with the Hoover Institution. Right now, we're looking into VTP funding by the Hoover institution. Nothing yet, but we've only started.

Posted by: Fishgrease | Nov 19, 2004 1:06:54 AM

Fishgrease:

You are right. VTP advocates AGAINST voter verified paper trail voting.
http://www.vote.caltech.edu/Reports/vtp_wp18.pdf

I totally disagree with their belief that machines can provide adequate auditing records. Frankly, their stance troubles me.

Their advocation for DRE may bias their analysis of the exit polls, it may not. But it raises doubts.

Anyway, on to the Berkeley study!

Posted by: Alex in Los Angeles | Nov 19, 2004 1:43:37 PM

Alex,

~~~ Their advocation for DRE may bias their analysis of the exit polls, it may not. But it raises doubts. ~~~

You've been more than fair with me. So, in kind, I must admit the possibility that the bias I see, Hoover institution or no, might not actually exist -- or at least might be due to some other factor I've not examined.

For instance, if the VTP committed to electronic voting early, thinking that the lack of a paper trail could be overcome validity-wise by digital signatures or some sort of checksum operation transparent to all involved, then we could just be witnessing some sort of institutional pride. And I cannot think of any institution that is immune to that.

Indeed! On to the Berkeley study!

Posted by: Fishgrease | Nov 19, 2004 3:05:05 PM

Fishgrease,

Your charges smack of the worst kind of McCarthyism. Is everyone affiliated with Stanford University suspect in your eyes, because Stanford also happens to house the Hoover Institution? Hoover is far from a "very deep, dark" organization: they are right out there, in the open, advocating conservative causes.

At to VTP advocacy of electronic audits, did anyone actually *read* the report? What it said is that a verified *audio* trail (a vocal recording that feedsback choices to the voter as they choose, resulting in a more accurate vote; then the recording is saved) would be superior to a paper record.

You may disagree with that approach, but it is far from sinister or conspiratorial.

Posted by: The Prof | Nov 24, 2004 8:05:20 PM

Prof,

Did you not read my very last post wherein I admit the bias I susupected might not exist or may have nothing to do with the Hoover Institution? When did McCarthyism become so flexible as that? And since you say my charges smack of the worst kind, might you provide an example of a better kind of McCarthyism?

I agree that the VTP paper on electronic audits was in no way "against" a paper record. I read it and your short summary matches mine. I don't agree with it regarding something so exotic as an audio record, but I'm not the one who posted that it was against paper records. Please review up blog.

Don't take any of benevolence in this post to mean that I suddenly like the VTP. I still think they came out in favor of electronic voting devices early and have since not published anything regarding the ridiculous situation we now find ourselves in where we have unauditable devices... a situation that can be traced back, in part, to a presentation given to the US House by the VTP in 2001. That presentation was delivered by a member of the VTP who is also a Fellow of the Hoover Institution, which, has very wide and undeniable links to the Bush administration. I'm not implying anything by stating those facts other than they are... facts.

If you used McCarthyism to mean that I find the present so called Black Box situation intolerable and am looking for those responsible, Democrat or Republican, conservative or liberal... guilty!

You need to show where I'm wrong in including the VTP among those, again at least in part, responsible. You need to show why their publishing a report to the contrary every time statistics show a problem with Florida's electronic voting isn't meant to color that situation as less intolerable... perhaps acceptable.

This country didn't just wander down the path toward unauditable voting devices. We were lead. Some of us now find that notable. If the Election Center, NASED, Diebold and groups like the VTP can justify leading us so then I'll we'll accept that also.

Posted by: Fishgrease | Nov 28, 2004 7:56:36 AM

eya FG

real good discussion at this level

of the voting issue. lil

nearsighted maybe but

well informed to

say the least.

Posted by: Snorky de fish boy! | Nov 28, 2004 5:46:05 PM

The Prof:

It seems you may be arguing semantics. VTP advocates/argues for an audio trail and against a paper trail. They do so both in the original report I cited and more directly in this one:
http://www.vote.caltech.edu/Reports/vtp_wp13.pdf

I'm not sure I understand your concern with my post. Maybe you mistakenly thought I did not know they argued for an audit trail? I did after all acknowledge "their belief that 'machines' can provide adequate auditing records."

Their stance against a paper trail troubles me, not because I question their motives, but because I disagree that paper audits can't be made to work and I disagree that any machine based audit trails, audio or otherwise, are truly voter verified. I also believe hand recounts ensure the greatest voter confidence.

Posted by: Alex in Los Angeles | Nov 29, 2004 1:51:17 PM

~~~ Their stance against a paper trail troubles me, not because I question their motives ~~~

Having re-read the article... they dance around standing against a paper trail and they should have just been truthful and come out against them. Audio trails! I DO suspect their motives, if I've not made that clear before now.

And they think we're going away. They think its all disolving. They think we'll forget. The VTP needs to think again about that.

Posted by: Fishgrease | Dec 12, 2004 5:41:13 PM

The comments to this entry are closed.