« Allocating Undecideds | Main | Exit Polls: Answering Your Questions »

November 02, 2004

Exit Polls: What You Should Know

*****************************

Looking for Information on Exit Polls for Election Day 2006?

See the newly updated, 2006 edition of this post now on Pollster.com!

*****************************

So sometime very soon, the traffic on certain web sites will hit astronomically high levels as the blogosphere goes in search of the latest leaked exit polls. The conventional wisdom on this is unshakable: The networks "know" who will win, but won't tell us. Lets take our own peek at results shared by those working at the networks today and get in on the secret.

Well, I hate to disappoint, but this site will not be a source of leaked "exits." However, I would like to take a moment and give you a bit of a reality check. Let me tell you a bit about what exit polls are and why you may want to take what you hear over the web with a giant grain of salt.

I have always been a fan of exit polls. Despite the occasional controversies, exit polls remain among the most sophisticated and reliable political surveys available. They will offer an unparalleled look at today's voters in a way that would be impossible without quality survey data. Having said that, they are still just random sample surveys, possessing the usual limitations plus some that are unique to exit polling (I also remain dubious about weighting telephone surveys to match them, but that is another story for another day).

A quick summary of how exit polls work: The exit pollster begins by drawing a random sampling of precincts within a state, selected so that the odds of any precinct being selected are proportionate to the number that typically vote in that precinct. The National Election Pool Exit Poll, which is conducting the exit polling for the six major networks today, will send exit pollsters to 1,495 precincts across the country.

One or sometimes two interviewers will report to each sampled precinct. They will stand outside and attempt to randomly select roughly 100 voters during the day as they exit from voting. The interviewer will accomplish this task by counting voters as they leave the polling place and selecting every voter at a specific interval (every 10th or 20th voter, for example). The interval is chosen so that approximately 100 interviews will be spread evenly over the course of the day.

When a voter refuses to participate, the interviewer records their gender, race and approximate age. This data allow the exit pollsters to do statistical corrections for any bias in gender, race and age that might result from refusals.

The interviewer will give respondents a 5 1/2 by 8 1/2 card to fill out that will include approximately 25 questions (see an example from the New Hampshire primary here). Respondents fill out the survey privately then put the completed survey in a clearly marked "ballot box” so they know their identities cannot be tracked and their answers remain confidential.

The biggest challenge to exit polls is logistical: How to transmit all the results to a central location quickly and accurately. In past elections, interviewers would take a 10 minute break every hour to tabulate responses. Interviewers would then call in tabulations at three approximate times during the day: 9:00 a.m., 3:00 p.m. and shortly before the polls close (disclaimer: I have no first hand knowledge of this year's procedures, which may be different).

Once the polls close, the interviewer will attempt to obtain actual turnout counts, and if possible, actual vote returns for their precinct. One of the unique aspects of the exit poll design is the way it gradually incorporates real turnout and vote data as it becomes available once the polls close. The exit poll designers have developed weighting schemes and algorithms to allow all sorts of comparisons to historical data that supports the networks as they decide whether to "call” a state for a particular candidate. When all of the votes have been counted, the exit poll is weighted by the vote to match the actual result.

So if this poll is so sophisticated, why can't we rely on the leaked mid-day "numbers” that will soon spread like wildfire across the web?

    1) It is still just a survey – Even when complete, an exit poll still has the same random variation as any other survey. NEP says typical state exit polls will have a sampling error when complete of +/- 4% at a 95% confidence level, and +/- 3% for the national exit poll. Even if comparable to the final numbers – which they are decidedly not – the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.

    2) The mid-day numbers do not reflect weighting by actual turnout – the end-of-day exit poll used to assist the networks in determining winners will be weighted by the actual turnout of voters at each selected precinct. The weighting will then be continuously updated to reflect turnout at comparable precincts. In the past, mid-day numbers have reflected a weighting based on past turnout, so the leaked mid-day numbers may tell us nothing about the impact of new registrants or the unique level of turnout this time.

    One point needs emphasis here: even in past elections, networks never called an election based on raw exit poll numbers alone. They were first weighted by a tally of the full day's turnout at each sampled precinct. This end-of-day data is (obviously) not available at 12 noon.

    3) Voting patterns may be different early in the day - People who work full time jobs typically vote more heavily before or after work. Even a perfect mid-day exit poll – and there is no such animal – may not be any better at picking a winner than the half-time scores in any given football game on Sunday. Also, despite what you may have heard on the West Wing, I know of no serious study showing a consistent Democratic or Republican tilt to the morning or evening hours (if anyone does, please email me).

    4) Early or absentee voting - As of last night, the ABC News survey estimated that 15% of all registered voters nationally had already cast absentee or early ballots. Obviously, these voters will not be available to interviewers standing outside polling places. To incorporate early voting, the National Election Pool is doing telephone interviewing in 13 states to sample the votes of those who voted early. Will these early votes be included in the mid-day leaked numbers? Who knows? I wouldn't count on it. (Good question, Andrew).

    5) They could be fictional - Both sides have huge armies of field workers sweating it out in the streets right now. Field workers have been known to find creative ways to boost the morale of their own troops or demoralize the other side. Might someone start a rumor by sending made up numbers to a blog? Ya think? After all, the guy most web surfers turn to for leaked exits likes to say that the information he provides is only 80% accurate. What are the chances he could be, excuse the technical term, making shit up?

    6) The people who do exit polls would rather you ignored them - OK, admittedly, that is a pretty wimpy reason, but they have a point. Exit polls provide a valuable resource for all of us. The will help us better understand who the voters are, why they vote the way they do and what the answers are to some of the debates that have raged for months that will not be resolved by vote returns alone. When someone leaks or broadcasts results of an exit poll (or telegraphs it by winking the way certain news networks tend to do about about 4 or 5 o'clock), calls are made to ban exit pollsters from polling places. That would be a very bad thing.

Listen, I understand human nature, and I'm not going to try to change it. We are all intensely curious about what is going to happen tonight, and most of us will find a way to peek at leaked exit polls at some point today. I just want you to know that those leaked exit polls really don't tell us much more about the outcome of the race than the telephone polls we were obsessing over just a few hours ago. Even if we wanted to call a race on unweighted, unfinished, mid-day exit polls alone (something the networks will not do), we would need to see differences of 10-15 points separating the candidates to be 95% certain of a winner.

So look at them if you must, but please, don't go plugging the numbers into spreadsheets and assume that your Electoral College "projections" have any special magic or scientific validity. Then don't. You might be better off flipping a coin to determine the outcomes of states like Ohio, Florida, Wisconsin, Iowa, etc.

I'll be back with more later in the day. If you have questions about exit polls that don't involve today's numbers, please sent them my way. You might also want to check the Frequently Asked Questions at the National Election Pool web site. See the jump page for offline sources this post.

P.S. I am leaving the comments section open, but with some very firm rules today. Absolutely NO POSTING OF LEAKED EXIT POLL NUMBERS. Anyone who chooses to ignore this admonition will have his or her message deleted and may be blocked from posting further. If I have to delete more than one or two messages, I will turn off comments for today. It may be the new Dad in me, but it’s my blog: don’t make me stop this car! ;-)

Embarrasing typos fixed 12:45 p.m. Points to anyone who caught "football dame." The MP needs some sleep.

Sources:
Merkle, Daniel M. and Murray Edelman (20020). "Nonresponse in Exit Polls: A Comprehensive Analysis." In Survey Nonresponse, ed. R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little, pp. 243–58. New York: Wiley. Election Polls, the News Media and Democracy, Paul J. Lavrakas and Michael Traugott (eds.), Chatham, N.J.: Chatham House Publishers.

Mitofsky, Warren J. (1991). "A Short History of Exit Polls. In Polling and Presidental Election Coverage, Lavrakas, Paul J, and Jack K. Holley, eds, Newbury Park, CA: Sage. pp. 83-99

----------------
A Special Request: Have you found this site useful? Please help me keep it alive by taking a few minutes to complete this brief (3-5 minute) about Mystery Pollster. Respondent identities will not be tracked and, as such, your participation will remain completely anonymous and confidential If you have any problems with the survey please email me. Click here if you're asking, "why a reader survey now?" Thank you!!

Related Entries - Exit Polls

Posted by Mark Blumenthal on November 2, 2004 at 11:30 AM in Exit Polls | Permalink

Comments

Hello, what kind of guidelines will the major news networks use in deciding when to report on exit data from particular states, and when to "call" the national election? I thought I remembered a few elections ago the networks had all agreed not to report any numbers until ALL the polls have closed.

Posted by: Chris Marstall | Nov 2, 2004 12:34:39 PM

"peek" to see results; not "peak" please.

Thanks for the good info.

Posted by: Amberlyn Morgan | Nov 2, 2004 1:00:01 PM

I heard a story on NPR this morning about the networks wanting to do a better job in their election coverage.

They talked about "wanting to get it right, not necessarily to get it first".

For example, ABC said they won't call any races that are within 1% using exit poll projections. CBS said its analysts will not be informed of other networks' calls on the outcome.

Any of the exit polls you see today at noon will be from the people who just want to be first with the scoop. Kudos for not falling into that mindset Mystery Pollster.

Posted by: johnzep | Nov 2, 2004 1:11:19 PM


Here's another problem with exit polls. If you report exit polls for 50 states each of which uses 95% confidence intervals then you have what is called by statisticians a serious "multiple comparisons" problem. That 95% confidence intervals only applies to that one poll viewed in isolation. If you want to have 95% confidence in ALL 50 polls, then you have to apply a stringent correction factor and margin of error goes up considerably.

Posted by: Brad | Nov 2, 2004 1:24:31 PM

Thanks for explaining exit polls. Very informative and useful.

Posted by: bncthor | Nov 2, 2004 1:33:34 PM

The late, much-lamented Chicago columnist Mike Royko always advised his readers to lie to the exit pollsters.

Posted by: Qoheleth | Nov 2, 2004 1:36:11 PM

I said it before, and I'll say it again, excellent site.

And thank you. With a newborn this is even more impressive. I am very appreciative of your effort.

Posted by: Eric | Nov 2, 2004 1:37:28 PM

I don't get the notion that the exit polls have a +/- 3% MOE nationally. I can see 4% MOE for a given state, but the sample set is so large nationally (150,000, given the stipulated numbers) that it's hard to believe that the MOE only goes down to 3%.

I can see that the larger states will impose larger errors nationally because, presumably, they have many fewer pollsters per voter than smaller states. But, of course, these effects should largely cancel out anyway, and I've got to believe that the huge numbers in the national sample will do a great deal to clamp down on errors.

I'd like to see how they come up with the 3% MOE nationally (do they even understand it to mean the same thing I am assuming they do?).

Posted by: frankly0 | Nov 2, 2004 1:50:14 PM

I'm sorry if you've already addressed this, but are pollsters attempting to correct for the large numbers of early voters?

Posted by: Ted Barlow | Nov 2, 2004 2:04:17 PM

How do you deal with people who voted before today, whether in person or w/ absentee votes. I assume you do polls to supplement exit voting? Do people in media/campaigns already have this data?

Posted by: Cableguy | Nov 2, 2004 2:26:30 PM

peek, not peak.

But we're all tired....

Posted by: JaBbA | Nov 2, 2004 2:38:28 PM

here's what josh marshall has to say about time of day voting:

"Exits come out in a several batches over the course of the day. Democrats, on average, tend to vote later in the day than Republicans. Not always, but that's the pattern, for fairly straightforward demographic reasons. And for that reason their exit poll numbers tend to get better over the course of the day. That was strikingly so in 2000. So if you see less than perfect numbers plastered around in the early afternoon, don't let that rattle you."

link: http://tinyurl.com/29yns

Posted by: m | Nov 2, 2004 2:39:51 PM

In addition to early voting, we should have a national holiday for federal election day. It might help avoid the crush periods. It would have they by product of also smoothing out the exit polls.

Posted by: WL | Nov 2, 2004 2:56:15 PM

Why does it matter that Josh Marshall says about exits? I say Republicans vote later in the day, simply for demographic reasons. Thus, my assertion is just as valid, isn't it? At least I'm not a (paid) flak.

Posted by: Dan | Nov 2, 2004 2:58:19 PM

Great summary on the exit polls. Can you tell us how the networks managed to screw them up so badly in 2000? And what have they done to correct that? It seems that since the exit pollers base their weighting on past elections, they will be even more unreliable this year than most given the large number of new or returning voters.

Posted by: London Lawson | Nov 2, 2004 3:17:41 PM

Great Job, MP. Your effort is much appreciated.

I must say that I was always under the impression you mentioned about early/late voting and was glad to hav ebeen set straight.

Posted by: Publius Rex | Nov 2, 2004 3:41:20 PM

Here's the google cache of a URL by John Mark Hansen analyzing the 'Democrats vote later, Republicans vote earlier' saw in relation to whether there should be uniform poll times.

Worth reading if only because he gives possible reasons for the shift, as well as presenting evidence both for and against it.

http://64.233.179.104/search?q=cache:EsBBXPNKyQsJ:millercenter.virginia.edu/programs/natl_commissions/commission_final_report/task_force_report/hansen_chap9_uniform.pdf+Democrats,+on+average,+tend+to+vote+later+in+the+day+than+Republicans.+&hl=en&start=3
(original is in PDF at http://millercenter.virginia.edu/programs/natl_commissions/commission_final_report/task_force_report/hansen_chap9_uniform.pdf)

Posted by: Sandy | Nov 2, 2004 3:50:50 PM

Small point regarding the exit poll methodology, but seems my old college stats class taught me that selection of every nth sample is not random. Truly random selection would be blocking off voters in same-sized groups, but then randomly picking from within each group. So, it might be voter #4 from the 1st block of 20, then voter #17 from the next block of 20, and so on. Like I said, pretty small point...doubt it would have much impact on the results.

Posted by: Dave Bailey | Nov 2, 2004 3:51:21 PM

"After all, the guy most web surfers turn to for leaked exits likes to say that the information he provides is only 80% accurate." I take that to be an allusion to Drudge. If so, it is a misrepresentation. He does not "like to say" that the "information" he provides is about 80% accurate. What he likes to say is that his forecasts / predictions (like who'll have the biggest box-office draw this coming weekend, etc.) are about 80% accurate.

Posted by: ELC | Nov 2, 2004 3:59:11 PM

The most shocking instance of poll failure I recall is the race between Feinstein and Huffington for senator of California in 1994. If you recall, Michael Huffington was a somewhat sleazy, somewhat lacking in intelligence, neophyte Republican who spent some $28MM of his own money to try to get elected.

The instant the polls closed, the networks all reported that Feinstein had won easily based on their exit polls. In fact, though, the race was extremely close, with Feinstein winning by about 1 percent.

The only possible interpretation is that people lied to exit pollsters. I wouldn't be surprised if people were embarrassed to have voted for Huffington, and then lied about it in the exit poll -- it wouldn't surprise me at all. If even 2 or 3 percent of the people surveyed did this, it would swing the exit poll considerably.

So -- be careful what you read!

Thad Beier

Posted by: Thad Beier | Nov 2, 2004 4:23:12 PM

Several thoughts.

First, as to your "#6". If they want us to "ignore" the numbers, there's an easy way to do that: don't realease them to ANYONE until the polls are closed.

What they really want is to pretend that they are an elite, and one of the ways they do that is by getting information no one else can have. I say, screw them.


How do the exit polls interact w/ GoTV efforts? I expect the GoTV people are having most of their effect later in the day ("Hi, I'm calling from the 'X' campaign, have you voted yet today?" Since you told their called last week you were voting for 'X'). So first of all this would be completely missed by the mid-day exit polls, no?

And what do they do when they hit 100 people an hour before the polls closed, because turnout was higher than they expected? Keep on interviewing?

Posted by: Greg D | Nov 2, 2004 4:37:20 PM

According to Matt "80%" Drudge, the early exits in 2000 showed Gore ahead 3pts in FL (wound up even) Even in Colorado (Bush eventually won by 9) and Gore ahead four in AZ (Bush won by 6) so if 2000 battleground states are to be believed, it sounds like Republicans vote in the afternoon, despite what Josh says.

Posted by: Tim | Nov 2, 2004 4:43:32 PM

Are you going to do anthing about trackbacks that show leaked numbers?

Posted by: Cube dweller | Nov 2, 2004 4:53:18 PM

On the National MOE: Who cares? This isn't a popular vote. Doesn't matter how close they get to the national vote, because that doesn't count for anything.

Posted by: John | Nov 2, 2004 5:48:13 PM

PUNCH ON George W. Bush
pineaple-ananás

Posted by: pineapple-ananás | Nov 3, 2004 1:51:00 AM

The comments to this entry are closed.