« The Freeman Paper Revisted | Main | More on the Berkeley Report »

November 22, 2004

WP's Morin on Exit Polls

Yesterday's Washington Post had a long story on the exit polls by Richard Morin, its director of polling. If you have followed the exit polling controversy with interest, it is absolutely positively a must-read. Of course, as a "self-important blogger" (see below), I am also duty bound to note that it plows a lot of familiar ground, confirming much of what you have already read here.

Since The Washington Post subscribes to NEP's data, Morin had full access to it on Election Day and has presumably seen the confidential reports by the exit pollsters to their clients. So he based the following on access to the hard data:

The sampling errors gave a boost to Kerry, who led in all six releases of national exit poll results issued on Election Day by the National Election Pool (NEP), the consortium of the major TV networks and the Associated Press that sponsored the massive survey project...

In the first release, at 12:59 p.m. on Election Day, Kerry led Bush 50 percent to 49 percent, which startled partisans on both sides. That statistically insignificant advantage grew to three percentage points in a late-afternoon release, where it remained for hours, even as the actual count began to suggest the opposite outcome. It was only at 1:33 a.m. Wednesday that updated exit poll results showed Bush ahead by a point.

Even more curious numbers were emerging from individual states. The final Virginia figures showed Bush with a narrow lead. Exit poll data from Pennsylvania, which was held back for more than an hour, showed Kerry ahead by nine percentage points. The actual results: Bush crushed Kerry in Virginia by nine points, while Kerry took Pennsylvania by just a two-point margin.

In a review of 1,400 sample precincts, researchers found Kerry's share of the vote overstated by 1.9 percentage points -- which, unhappily for exit pollsters, was just enough to create an entirely wrong impression about the direction of the race in a number of key states and nationally.

Morin also confirms that previous years showed a similar, though typically smaller Democratic skew.  He ads one important new wrinkle -- 1992 had a similar skew: 

It's hardly unexpected news that the exit polls were modestly off; exit polls are never exactly right. The networks' 1992 national exit poll overstated Democrat Bill Clinton's advantage by 2.5 percentage points, about the same as the Kerry skew. But Clinton won, so it didn't create a stir. In 1996 and 2000, the errors were considerably smaller, perhaps just a whiff more Democratic than the actual results. That suggests to some that exit polls are more likely to misbehave when their insights are valued most -- in high-turnout, high-interest elections such as 1992 and this year [emphasis added].

Morin also answered a question that comes up repeatedly:

Perhaps the Democratic skew this year was the result of picking the wrong precincts to sample? An easy explanation, but not true. A post-election review of these precincts showed that they matched the overall returns. Whatever produced the pro-Kerry tilt was a consequence of something happening within these precincts. This year, it seems that Bush voters were underrepresented in the samples. The question is, why were they missed? [emphasis added]

This piece would have been stronger without the usual gratuitous slap at bloggers (though this one is a bit back-handed):

It's also time to make our peace with those self-important bloggers who took it upon themselves to release the first rounds of leaked exit poll results...but rather than flog the bloggers for rushing to publish the raw exit poll data on their Web sites, we may owe them a debt of gratitude. A few more presidential elections like this one and the public will learn to do the right thing and simply ignore news of early exit poll data. Then perhaps people will start ignoring the bloggers, who proved once more that their spectacular lack of judgment is matched only by their abundant arrogance.

Oh well....I've said my piece on this issue already. At least Morin says bloggers are "his new best friends." Name calling aside, the article remains worth reading in full.

UPDATE:  Morin had much more to say about the exit polls in this online chat -- even some nice words for a certain blogger.  Who knew? 

Related Entries - Exit Polls

Posted by Mark Blumenthal on November 22, 2004 at 01:27 PM in Exit Polls | Permalink

Comments

It is rather interesting that in 1992 the exit polls overstated the vote of the more "left-wing" party and understated that of the more "right-wing" party in both the US and the UK. I'm wondering if that is part of a more general pattern--that people who vote for right-wing parties are marginally less likely to admit it to major media exit pollsters (who they rightly or wrongly see as representing the liberal "chattering classes" etc.)

Posted by: David T | Nov 22, 2004 5:04:18 PM

Given the Bush Administration's, Karl Rove's and the Florida Election Commission's reputations for dishonesty and dirty tricks, doesn't it seem reasonable to keep deliberate vote fraud as the the working hypothesis to explain Florida's puzzling election results until we have some truly overwhelming evidence to the contrary?

Posted by: Peter Gaffney | Nov 22, 2004 5:42:51 PM

Can we finally, finally, finally say that there needs to be some type of investigation here. There are just too many discrepancies too much in one direction. Has anybody seen the post on Josh Marshall's Talking Points about the Ukraine elections? The Ukranians take to the streets and threaten strikes on basically this same information. That we lived to see the day when Ukranians put us to shame in their care for demcoracy.

Posted by: Wilbur | Nov 22, 2004 6:32:24 PM

"That suggests to some that exit polls are more likely to misbehave when their insights are valued most -- in high-turnout, high-interest elections such as 1992 and this year."

Can you post a historical survey of exit poll accuracy with real numbers?

If inaccuracy in fact correlates to high-turnout, high-interest elections, vote spoilage may play a role. I have not found any literature that describes the role of vote spoilage in exit polling and this seems fundamental when votes spoilage exceeds millions of votes in every presidential election.

My admittedly weak reasoning:
*High turnout, greater number of low income/minority voters
*Greater use of low quality vote equipment found in low income/minority voters
*Millions of spoiled votes affecting disproportionally low income/minority voters.


And frankly, shouldn't exit poll analysis incorporate data on spoiled votes? Respondents would not know their vote was spoiled when filling out the exit poll.

Posted by: Alex in Los Angeles | Nov 22, 2004 7:13:17 PM

> Given the Bush Administration's,
> Karl Rove's and the Florida Election
> Commission's reputations for dishonesty
> and dirty tricks, doesn't it seem
> reasonable to keep deliberate vote fraud
> as the the working hypothesis to explain
> Florida's puzzling election results until
> we have some truly overwhelming evidence
> to the contrary?

No.

Posted by: Alan | Nov 22, 2004 8:31:17 PM

To expand on Alan's comment:

No. It doesn't seem reasonable in the least.

---

Let me say it as slowly and simply as possible:

There may have been some old-style quasi-legal voter suppression fraud in this election.

There may have been some limited new-fangled electronic no-paper-trail fraud in this election.

But (and here comes the crucial part), the skewed exit polls play zero part in proving, or even indicating, either kind of fraud.

If you think there has been fraud, find some other way of showing it. The skewed exit polls have nothing to do with it. You are barking up the wrong tree. You are wasting the time and attention of those trying to find, demonstrate, and prove fraud. You are walking into a dead end. You are trying to prove the theory of evolution by commenting that apples are delicious - the two are unrelated.

Posted by: Petey | Nov 22, 2004 8:48:43 PM

Another obvious test for exit polls:
How did the exit polls do in predicting the other races they were polling?

Just another question to keep in mind when the NEP report or raw data are released, because unless Dr. Freeman also has that data it probably doesn't exist in public.

Also, I just must repeat that the false choice between fraud in vote count or bias in exit poll ignores vote spoilage as a factor. How much vote spoilage contributes to exit poll discrepancies has not been looked at in this blog yet.

Posted by: Alex in Los Angeles | Nov 22, 2004 9:55:33 PM

Alex asked...

"Can you post a historical survey of exit poll accuracy with real numbers?"

If only I had one. Unless there is a study out there I'm not aware of, the Morin article discussed here is about as good as it gets.

He also asked:

"Another obvious test for exit polls: How did the exit polls do in predicting the other races they were polling?"

That's a *really* good question. Unfortunately, I'm not sure Freeman's source bothered to grab election night results for the exit polls for other races. Blogs obsessed over the leaked results for President, but I don't remember seeing any that posted other races.

Alex has also asked...again and again and again..about vote spoilage. I hear you, Alex. If only I had more hours in the day! I'll try to write something on this soon.

Posted by: Mark Blumenthal | Nov 22, 2004 11:02:42 PM

Petey and all,

If exit polls play zero roll in determining fraud do you believe the Ukranian president was right to call out the tanks? Do you believe the United Nations and the United States (especially Richard Lugar) should be wearing tinfoil hats because they believe there were irregularities in the Ukraine elections? Do you believe the two other former Soviet Republic elections that were overturned should be given back to the rightful winners? Do we need to send in the troops? Just asking.

Posted by: Wilbur | Nov 23, 2004 8:10:38 AM

Mark,

Sorry, but squeaky wheels and all that. Your work is awesome and each post really is a week's worth of work at least. Indispensable service to the american public.

Morin argues this election will teach the public to properly value/discredit exit polls, but IMO your articles are the true reason the American public will finally understand the value and limitations of our electoral system. Self important bloggers? More worthy of scorn are useless journalists that don't respect their readers.

Thank you.

Posted by: Alex in Los Angeles | Nov 23, 2004 11:40:16 AM

Care to comment on www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=72483&mesg_id=72483

Posted by: John Kesich | Nov 23, 2004 2:58:56 PM

Something that has bothered me is the use of the term "exit poll" to describe both the raw data and the projections posted. Surely professionals don't use such sloppy terminology, what are the proper terms?

Posted by: John Kesich | Nov 23, 2004 5:08:08 PM

--Early Exit Polls (afternoon releases possibly unweighted)
--Final Exit Polls (released at poll closing weighted by race, gender, age, and turnout)
--Corrected/Revised/Reweighted Exit Polls (weighted by reported vote counts). The lack of a clear term for this type of exit poll probably led to Caltech/MIT's confusion.
--Exit Polls raw data (by precinct, may or not be weighted by race, gender, age, and turnout)

These terms aren't necessarily academic but they've been used on this blog.

Posted by: Alex in Los Angeles | Nov 23, 2004 6:07:21 PM

Wilbur,

Well, let's consider reality.

1: The average of the polls leading up to the election had Bush up, and had late breakers tending towards Bush.

2: The US Exit polls have typically been biased against Republicans.

3: From what I've read, once the exit polls were "reweighted" using the turnout numbers from the day, they showed a Bush victory.

4: IIRC, the exit polls didn't continue until teh polls were closed, so late voters (you know, Republicans voting after they got off work) weren't adaquately represented.

5: In this election cycle, as in past election cycles, there's been one party favoring measures to prevent vote fraud 9such as election monitors), and one party fighting to make vote fraud easier (such as opposing requirements that people show ID in order to vote).

In every case I know of, it's been the Republicans who are anti-fraud, adn the Democrats who are pro-fraud.

So seeing Democrats make up claims of fraud now just sounds like "projection" to me.

6: The Ukranian Exit polls found a wide gap, the US polls did not. Wide gaps are more relevant than 2 point ones.

Posted by: Greg D | Nov 23, 2004 8:07:13 PM

I've been in contact with Morin. He did not have access to ALL the data on election day, but he did have access to some. This did surprise me.

I've analyzed the exit polling data based on a data made available by a working paper prepared by Dr. Freeman of UPenn. I will be posting on this issue very soon.

Posted by: Rick Brady | Nov 24, 2004 12:53:50 AM

Greg D wrote:
>
> 1: The average of the polls leading up to
> the election had Bush up, and had late
> breakers tending towards Bush.

According to the Electoral Vote Predictor, which summarizes exit polls, Kerry was leading Bush 283 to 246 on Oct 31st and 298 to 231 on Nov 1st (on Nov 2nd it shows a Kerry lead of 262 to 261, but I don't know if that was done before or after voting had begun):

http://www.electoral-vote.com/nov/oct31.html
http://www.electoral-vote.com/nov/nov01.html
http://www.electoral-vote.com/nov/nov02.html

> 2: The US Exit polls have typically been
> biased against Republicans.

Can you cite a source supporting this assertion?

> 3: From what I've read, once the exit
> polls were "reweighted" using the turnout
> numbers from the day, they showed a Bush
> victory.

If there was fraud any kind of "reweighting" is highly suspect. We need to know what the raw exit polls showed, on a county by county level, and what the exact error margins were. Even then the numbers will be suspect since there weren't any other exit polls done by other independent organizations.

> 4: IIRC, the exit polls didn't continue
> until teh polls were closed, so late
> voters (you know, Republicans voting
> after they got off work) weren't
> adaquately represented.

And how about those Democrats in poor neighborhood who waited ten hours to get to vote because they had too few voting machines? http://www.freepress.org/columns/display/3/2004/990

> 5: In this election cycle, as in past
> election cycles, there's been one party
> favoring measures to prevent vote fraud
> 9such as election monitors), and one
> party fighting to make vote fraud easier
> (such as opposing requirements that
> people show ID in order to vote).

And there's been one party that's been intimidating and disenfranchising voters on a massive scale and one which has been fighting to have everyone vote: http://en.wikipedia.org/wiki/2004_U.S._Election_controversies_and_irregularities

> In every case I know of, it's been the
> Republicans who are anti-fraud, adn the
> Democrats who are pro-fraud.

Then you don't know very much. Read the wikipedia article above to learn about thousands of cases which show the opposite.

> 6: The Ukranian Exit polls found a wide
> gap, the US polls did not. Wide gaps are
> more relevant than 2 point ones.

There is a gap of sometimes up to seven percentage points in some areas of the country this election.

Posted by: aaa | Nov 24, 2004 3:02:06 AM

Richard Morin wrote:
>
> In the first release, at 12:59 p.m. on
> Election Day, Kerry led Bush 50 percent
> to 49 percent, which startled partisans
> on both sides

Not really. Polls had shown a Kerry lead in the last two days leading up to the election:

http://www.electoral-vote.com/nov/oct31.html
http://www.electoral-vote.com/nov/nov01.html

And all along leading up to Nov 2nd the pundits were opining that the race was "too close to call"... so to say that a 50 to 49 percent Kerry lead "startled" anyone is simply revisionist.

> Whatever produced the pro-Kerry tilt was
> a consequence of something happening
> within these precincts.

Or maybe it's a problem with the exit polling methodology. Until the public has all the raw data and knows the methodology behind them we'll never know. Even then the exit polls will be suspect, as there won't be other exit polls, done by independent organizations, to check them against. We'll just have to take the NEP's word for it, which may just not be good enough in an election which reeks of fraud.

> This year, it seems that Bush voters were
> underrepresented in the samples.

Or that there was fraud. Why doesn't he admit this possibility?

> The question is, why were they missed?

No. The question is: were Bush voters missed or was there fraud?

Posted by: aaa | Nov 24, 2004 3:13:28 AM

Yes Greg D let's consider reality. The exit polls in the Ukraine had the opposition ahead by three. The final results had the incumbent up by three. Not even quite as bad as the Ohio differential and no different from a number of other states (remember that the presidential election is actually fifty individual elections).

Posted by: Wilbur | Nov 24, 2004 10:39:05 AM

I followed the Morin link and found a reference to NEP polls of absentee and other early voters.

Does anyone have additional information this? I am in the middle of a research project on early voting and would really appreciate any references to polling data on early voters.

Posted by: The Prof | Nov 24, 2004 7:55:24 PM

General comment:

wikipedia is a worthless "resource", because anyone can change its contents whenever they want. Referencing it for anything contraversial is stupid, and a waste of time.

Posted by: Greg D | Nov 24, 2004 11:30:55 PM

Wilbur wrote:

The exit polls in the Ukraine had the opposition ahead by three.

http://www.dw-world.de/dw/article/0,1564,1403660,00.html

"Exit polls on Sunday indicated Yuschenko, who wants to steer the country out of Russia's shadow and towards Europe, had won 54 percent of the vote compared with the 43 percent of his pro-Russia rival, Prime Minister Viktor Yanukovich, according to the independent Razumkov Ukrainian Center for Economic and Political Studies and the Kiev International Institute for Sociology (KMIS)."

54 - 43 = 11

http://www.nj.com/news/ledger/index.ssf?/base/news-18/1101113463221380.xml

"The exit poll, conducted under a program funded by several Western governments -- including the United States, showed Yushchenko with 54 percent of the vote compared with 43 percent for Prime Minister Viktor Yanukovych. A second exit poll, however, showed Yushchenko's margin much smaller at 49.4 percent to 45.9."

[snip]

"The poll sponsored by Western governments was based on responses by more than 20,000 voters at polling places throughout the country, according to the Democratic Initiatives Foundation, which coordinated the effort.

"The second poll, conducted by Ukraine's Sotsis and Social Monitoring, was based on interviews with 13,000 voters, the Interfax news agency reported. The second poll was conducted through face- to-face interviews rather than the first poll's method of having respondents fill out anonymous questionnaires.

"In the first-round vote held Oct. 31, the face-to-face method was criticized as potentially intimidating for voters who worried about expressing opposition to Yanukovych."

So, Wilbur, are you ignorant (hard to beleive, since you used the phrase "latest exit poll" in an attempt to ignore the more believable poll), innumerate, or dishonest?

Posted by: Greg D | Nov 24, 2004 11:38:09 PM

Oh, BTW, what Wilbur tried to pull here is another example of what's so totally wrong with Democrats today.

He's upset about our election, and is responding with NO concern about the costs of his response. Does it matter if the people of Ukraine have their votes stolen, if then remain shackled with a corrupt and authoritarian government? No, not if he can score a cheap shot of Republicans by ignoring what's happening to the Ukrainians.

Was there anything the Democrats weren't willing to do (other than adopt reasonable policies that might appeal to a majority of American voters) in order to try to win the 2004 election? Let's consider a few of the things they were willing to do:

Ally with the al Qaeda (they feared capturing Osama, rather than supporting it. They supported OBL talkign points, and gave him new ones).
Ally with Sunni terrorists and Baathist dead-enders.
Support corruption (see UN Oil-for-Bribes).

Now we see them supporting Putin's desire to keep Ukraine subservient to Russia.

A Party that does not love America does not deserve Americans' votes. A Party that loves America puts America's welfare above the Party's welfare.

Today's Democrats put nothing above their desire for power. Which is why they're not qualified to have any.

Posted by: Greg D | Nov 29, 2004 5:25:50 PM

The comments to this entry are closed.