A commenter asked last week, "why are exit polls so much more accurate in Europe?" This is a question worth considering, because all surveys are not created equal. Differences in sample sizes, response and coverage rates and the experience and training of interviewers can tell us a lot about the potential for survey error.
I cannot claim personal expertise in European exit polls, but a Google search quickly turned up a rather contrary opinion published earlier this year by the ACE project (an acronym for Administration and Cost of Elections, a joint project funded by the UN and the US Agency for International Development):
[Exit poll] reliability can be questionable. One might think that there is no reason why voters in stable democracies should conceal or lie about how they have voted, especially because nobody is under any obligation to answer in an exit poll. But in practice they often do. The majority of exit polls carried out in European countries over the past years have been failures [emphasis added].
Presumably, the newfound belief in the accuracy of European exit polls comes from Steven Freeman’s evolving paper, the Unexplained Exit Poll Discrepancy. Freeman placed great emphasis on the "highly reliable" results from Germany. He showed results from exit polls conducted by the Forschungsgruppe (FG) Wahlen on behalf of the ZDF television network that were off by only 0.26% over the last three national elections.
Freeman also cited another "consistently accurate" survey, this one conducted by student volunteer interviewers in Utah. In 2004, according to Freeman, the Utah Colleges Exit Poll came within 0.2% of the actual result for the 2004 presidential race. "Consistently accurate exit poll predictions from student volunteers," Freeman concluded, "including in this presidential election, suggest we should expect accuracy, within statistical limits, from the world’s most professional exit polling enterprise."
Four weeks ago, after I posted my original critique of his paper, Freeman called seeking further input. My advice included the strong suggestion that he check on the methodologies used for the German and Utah polls before implying that the NEP surveys were comparable. As of this writing, Freeman’s paper still lacks any reference to basic methodological information regarding the German and Utah exit polls.
I made some inquiries about both, which I summarize below. For the sake of comparison, let’s begin with the methodological details of the National Election Pool (NEP) exit polls:
NEP Exit polls
- State exit polls sampled 15 to 55 precincts per state, which translated in 600 to 2,800 respondents per state. The national survey sampled 11,719 respondents at 250 precincts (see the NEP methodology statements here and here)
- NEP typically sends one interviewer to each polling place. They hire interviewers for one day and train them on a telephone conference call.
- The interviewers must leave the polling place uncovered three times on Election Day to tabulate and call in their results. They also suspend interviewing altogether after their last report, with one hour of voting remaining.
- The response rate for the 2000 exit polls was 51%, after falling gradually from 60% in 1992. NEP has not yet reported a response rate for 2004.
- Interviewers often face difficulty standing at or near the exit to polling places. Officials at many polling places require that the interviewers stand at least 100 feet from the polling place along with "electioneering" partisans.
German Exit Polls (by FG Wahlen)
Dr. Freeman’s paper includes exit poll data conducted by the FG Wahlen organization for the ZDF television network. I was able to contact Dr. Dieter Roth of FG Wahlen by email, and he provided the following information:
- They use bigger sample sizes: For states, they sample 80 to 120 polling places and interview 5000 and 8000 respondents. Their national survey uses up to 22,000 interviews.
- The use two "well trained" interviewers per polling place, and cover voting all day (8:00 a.m. to 6:00 p.m.) with no interruptions.
- Interviewers always stand at the exit door of the polling place. FG Wahlen contacts polling place officials before the election so that the officials know interviewers will be coming. If polling place officials will not allow the interviewers to stand at the exit door, FG Wahlen drops that polling place from the sample and replaces it with another sample point.
- Their response rates are typically 80%; it was 83.5% for the 2002 federal election.
- The equivalent of the German of the US Census Bureau conducts its own survey of voters conducted within randomly selected polling places. This survey, known as "Repräsentativ-Statistik," provides high quality demographic data on the voting population that FG Wahlen uses to weight their exit polls.
Dr. Roth also added the following comment: "I know that Warren Mitofsky’s job is much harder than ours, because of the electoral system and the more complicated structure in the states."
- The Utah poll typically consists of 90 precincts and between 6,000 and 9,000 respondents. Compare that to the NEP exit poll for Utah in 2004, which sampled 15 precincts and 828 respondents.
- Interviewers are student volunteers from local universities who attend an in-person training seminar.
- They assign eight student interviewers to each precinct working in two shifts of four each. The larger number of interviewers allows coverage of all exits at each polling place all day long.
- The response rate has typically been 60%, although the 2004 survey attained a response rate of roughly 65%.
In short, there are sound methodological reasons why the German and Utah exit polls typically obtain more accurate results: They do more interviews, attain better coverage and better response rates and use arguably better trained interviewers.
Of course, the lower response rates and coverage problems, in and of themselves, do not explain why the NEP exit polls had a slight Kerry bias this year. If the error was in the surveys, then something made Kerry supporters either slightly more cooperative or more likely to be interviewed. The fact that response and coverage rates were lower did not cause the error, it just made the potential for error that much greater.
12/20-corrected misspelling of Wahlen in original
Yes- but what about the Diebold counting machines!!!! Also – prove the existence of this design effect phenomenon! Who cares if you’ve completely debunked the methods and logic behind Dr. Freeman’s paper if you haven’t given us something better to use!!! Until you give us the right odds, we have to assume the 1:650,000 is accurate!!!! Who do you think you are? Do you really think we will let you get away with thinking for yourself!
Okay that was a bit mean, but I just read TruthIsAll’s “rebuttal” to one of your posts and I’m a bit frustrated…
For a good laugh that lasts only as long as you think it is a joke, read this:
http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=203×178839
Thanks — yet again — for the research.
Any insight into why the Utah survey gets a higher response rate than the NEP? One answer is that the NEP gets high response rates in Utah, so there really is no difference. The other is that there is something in the way people are approached that makes them more likely to participate in the Utah survey. If the latter, I strongly suggest NEP take lessons.
The other comment is that I strongly suggest that non-partisan national exit polls be conducted in 2006 and 2008 that are modeled on the more successful methods. What would such a nationwide survey cost, approximately?
Observer, you raise a good question that I hope MP will grace us with his presence and haszard a guesstimate.
I would have loved to have been a fly on the wall in the negotiations between the NEP members, Mitofsky, and Lenski.
From what I know from my prof, who owns a small polling company in SD, a pollster asks a client what level of accuracy they want (basically a function of how many precincts they will have to sample and how many samples at each precinct). The client often wants something outrageous like +/- 1% (my prof deals with rookies) and they come back with a cost required for that degree of accuracy. Then, the client backs way off and settles for something like 4 or 5% (national public opinion polls are typically taken at +/- 3%, which requires ~1100 respondents, but state opnion polls are often done at +/-4%, which requires ~900 respondents).
So basically, assuming sound methods, accuracy is a largely a function of money and therefore it would be interesting to find out how much “accurate” exit polls (i.e. <+/-1%) in at least the battle ground states would cost. Care to chime in MP?
The silence over at DU (actually from TIA) is deafening….
http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=178839&mesg_id=179733&page=
The arrogance of this guy is astounding…
http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=178839&mesg_id=179743&page=
Arrogance or lack there of proves nothing, RB. We’re all trying to get to an understanding here. But your concise thoughts about what you think is wrong with TIA’s arguments or where TIA and Mystery Pollster might be crossing wires would be appreciated.
cmg,
What is wrong with TIA’s arguments. You could write a book, but for starters the guy MUST read up on the standard error associated with exit polls v. simple random samples. Actually, all he has to do is read MP. I’ve posted on the subject as well.
If you realize his error on this point alone, all the rest of his arguments don’t need refuation – they refute themselves.
I suspect MP won’t get into an argument with TIA because it isn’t a match.
I once saw a debate between Alan Dershowitz and Anne Coulter. Poor Alan, I disagree with just about everything he says ideologically, but he sure is brilliant. Anne??? Well, she churns my stomache and I suppose she could be smart, but she sure doesn’t show it. Why did he debate her? I bet he won’t do that again. It was quite silly really.
Once again I ask:
Shouldn’t we be exit polling Ohio now (say 4,000 – 5,000 voters) to get random, not clustered, exit poll data, accurate to .05%?
davidgmills, did you see my comment in the post below re this issue? There is a bandwagon effect post election. There was a huge spike in the number of people who call themselves Republicans shortly after the election. Read my other comment and you’ll see why I think post-election polling of who someone voted for would likely only confirm the victor.
David: only if you are Bob Barker and there are three candidates. Then you ask everyone to pick one, and tell them which of the other other two didn’t win and ask if they want to switch.
Unfortunately, the extra information matters.
Observer, Rick, all:
Sorry to have missed these comments. Home with this amazingly brutal cold virus I picked up from my toddler. Go figure.
As for why Utah gets a higher response rate, I can only speculate. I’d guess their connection with local universities, rather than the big networks is a big plus. There is research on telephone response rates that shows University sponsored surveys get the best response rates, followed by major media brands, followed by everyone else.
As for the cost issues, all I know is the $10 million total figure that gets thrown around, but I have absolutely no idea if that’s accurate. Keep in mind that up until 1988 (if memory serves) each network did its own exit polls. The networks decided to combine forces, first with VNS and this year NEP, in order to keep those costs down.
Judging from the ratings, I’d guess the news divisions are under more pressure than ever to hold down these expenses, rather than devote more resources to them.
As for the TIA post, I’d agree with Rick that the primary disagreement, as I follow it, is that he doesn’t “believe” that cluster samples have more random error than simple random samples. That is a little like not believing in gravity. Beyond that I’d say my post speaks for itself.
Thanks for reading in December!
Mark
John:
I guess you think Americans have no shame.
I just don’t really think this is a subject that people would lie about. They may refuse to tell you, they may tell you it is not your business, but if they give a name, I really do not believe many would lie.
Frankly, I think lying would cause less of an error than voter mistake.
davidgmills,
How then do you explain the surge in people who affiliate with the Republican party when compared to a similar poll of party membership pre-election? Read the link in my comment to a couple posts below this one.
If you cannot explain the surge in Republican Party affiliation, then I suspect you will have to assume many of these people are lying (or are simply bandwagon party affiliators). I don’t know, but you’re a lawyer, maybe you could pull a group of finaciers and raise $500,000 and hire Zogby or someone to poll 30,000 Ohio voters to test your hypothesis. That should give you a +/- 0.5% CL.
Of course, the chance of a Type I error would be large, so don’t be quick to reject that null!
Rick:
What is type 1 error and why would it be large?
What has surge in Republican affiliation got to do with anything?
Mostly it was at the expense of independents.
Democrats stayed about the same.
By the way, Truth is All, a mathmatician, thinks MP knows little about math. For what it is worth. And the beat goes on.
Pick your scientist. it seems like they can find fault with another’s work.
Rick:
One more thing.
If we took a poll of people and gave them a choice of being corporatists or anti-corporatists, who do you think would win?
What would you be?
On the contrary, David, my assertion is that people behave in complex but predicatable ways which include some percentage that will switch and some that will hold fast, depending on intangibles such as perceived risks and gains.
BTW, evil corporations already use the tendency of people *not* to switch in game shows when they should to keep their costs down.
If you want to beat corporations, you should be so unpredictable and inconsistant they go broke trying to figure you out. Lying in exit polls is a great way to do it.
davidgmills, a type I error is rejecting a null hyopthesis as false when the null hypothesis is actually true. One way to think of a type I error is putting an innocent person in jail.
In your case, the null is that if the election result was accurate, there should not be a statisitically significant difference between the telephone poll and the election result.
In other words, assume you take your poll of Ohio and it shows that Kerry’s actual election result was “significantly” less than what your poll showed it should be. This conclusion could be spurious. People could lie about having voted (meant to, whoops, but the lines were soooo long at my precinct – however, I would have voted for Kerry if the lines weren’t so long). There could also be other reasons, but since Bush won the election by almost 3%, if your poll (at +/-0.5%) showed that Kerry should have actually won the election, then you are right – I would really suspect something is up. Your poll would likely make MAJOR news and would perhaps be replicated by other indepedent sources before people (with authority and power) started screaming fraud. However, my point is that there is the possibility of committing a Type I error in rejecting any null.
There is also the possibility of committing a tpye II error with your poll, which is failing to reject a false null hypothesis. One way to think of a type II error is letting a guilty person go free.
This goes to my comments about bandwagon voters. Suppose Kerry really did win OH, but many Kerry voters are embarassed they voted for a losing candidate and lie to your pollster. Again, who knows? You have to be aware of this possibility.
Frankly david, if someone were to poll 25,000+ Ohio voters, I’d be very interested. This wouldn’t cost $500K either (I don’t think – possibly in the tens of thousands). However, don’t cut corners on price by hiring someone with little experience (i.e., is someone who is not nationally recognized authority on polling).
“Shouldn’t we be exit polling Ohio now (say 4,000 – 5,000 voters) to get random, not clustered, exit poll data, accurate to .05%?”
David, I don’t have my book here, but I think you need more than 25K persons to get +/-)0.5% at 95% CL, and even more if you want to be 99% confident of your results.
To get +/- .05% would probably require a new election (hundreds of thousands), but I think that was a typo :-).
Once you get around +/-1%, the number of samples required to tighten this confidence interval increases almost exponentially. That is why for cost reasons, most national polls do +/-3%. To get +/-2% requires almost 2X the samples. I wish I had my book here to be precise about all of this. I’ll try to report back later.
Rick, I think a shorter answer to David’s “Shouldn’t we be exit polling Ohio now” is the next time we can do that is 4 years from now, no matter how much money we spend or how big the sample. It’s not an exit poll if it’s not a presidental election and people are exiting the polling place.
John, I know what you are saying, but then you would have to admit that even portions of the exit polls themselves are not really exit polls because they rely on phone calss to early and absentee voters. check out the nep methods statements. dan merkle of abc told me that even these phone samples are not entirelyrandom and have a (smaller) design effect. holding 16 mo old while typing yeah!!!
Rick and John:
I still don’t think people would intentionally lie about who they voted for. When people ask you who you voted for four, eight, twelve, years ago, do you lie about it? I may have forgotten who I voted for sixteen or twenty years ago, but I don’t lie about it. Heck, I voted for Reagan; twice I think. What was I thinking?
People may not want to divulge who they voted for, but I don’t think it is something people lie about. As a lawyer, I see many people sworn to tell the truth. Contrary to popular belief, I think most try to tell the truth. They may slant it their direction, but most generally will not lie to a direct yes or no question.
So let’s have post-election polling if you think the term exit-poll is not appropriate.
By the way .05% was not a typo. That was my target MOE if costs permitted. But I have no idea how many people it would take to get to that MOE. I wouldn’t have thought it would take more than 4,000 – 5,000, but obviously I am wrong. I would like to know, though. I think you could do it cheaply with trained college students though.
About the evil corporations. I am not a socialist. I am a capitalist. But my idea of capitalism is small business not large corporations. Small businesses have a very difficult time competing against corporations that are in bed with the government. Small businesses spread out economic power.
Okay david, to give you a sense of how slope of the non-linear function involved, here is what the textbook says (surveyed populations over 100,000):
+/-5% = 664 samples
+/-4% = 1,037 samples
+/-3%= 1,844 samples
+/-2%= 4,148 samples
+/-1%= 16,590 samples
Okay, now to achieve +/- 0.05% at 99% CL, you have to use a formula: n=((2.575*.5)/.0005)squared
This means you have to take a sample of 6,630,625 to achieve your parameters. Of course there weren’t even that many voters in Ohio, so this is not feasible, but you could obtain that high degree of precision if you polled almost 7 million voters nationally.
But why bother? Why not take +/- 1%? Bush won by 3%, that means that at 99% CL, a significant discrepancy would turn some heads… A non-significant discrepancy should quiet the conspiracy theories as well (but remember the potential for Type I and II errors!)
BTW – I wouldn’t use grad students. They are known to screw up a lot of things (he hem). Stick with the pros.
Rick:
Thanks. A lot more samples than I would have expected. But I guess polling 16,000 people is not likely to break the bank. In my view it would be money well spent.
There is something in law called the “harmless error” rule. It is the rule that governs all appeals. It essentially says that unless you can show that the mistakes or errors by the judge or jury were likely to cause the rendition of an improper verdict, the appeal has no merit. Most appeallate cases are lost because the appellate courts decide that the mistakes were essentailly harmless.
By analogy, how does one show that the voting mistakes, error and/or fraud, likely caused the election to be won by the wrong man? I suggest you can’t, unless you can have a real valid means of proving that the errors, mistakes and/or fraud likely caused the wrong person to win. That’s why I believe that if the Democrats are serious about their investigation of Ohio, they also need to have some means of showing that it was likely these errors, mistakes and/or fraud cost Kerry Ohio. Right now, they can’t make such a showing. But a poll like this one, might be able to do it, if Kerry wins the poll by the margin the exit polls predicted he would have.
Of course, if a poll like this one showed Bush to be the winner, that would make most of the arguments moot, except maybe those arguments for not having enough voting machines. Those arguments are valid to make changes for the next election, but of course, would have nothing to do with the validity of this one.
Just remember david, should you try this survey, it’s not as easy as hiring a bunch of grad students and “randomly” calling people from a phone book. The survey would have to be based on a stratified random sample, which is different than a simple random sample. The stratified random sample is necessary because you would want to ensure your sample adequately represents of the geography and population for key demographic variables (race in particular and perhaps age as well). This may require weighting and correcting procedures. Although, with such a high sample size, I’m not sure how much weighting would be necessary. This gets a bit tricky… Not for amateurs or grad students. Perhaps MP would be willing to take your money?
MP who? Seriously, I think it should be done by Zogby, Harris, and Gallup, each doing a third of the survey, even if they are corporations.
I am not going to try this survey myself; my hope is that concerned democrats would do it and put an end to this speculation one way or another.
Rick,
Why would you stratify David’s poll?
I thought stratifying was useful only when you wanted demographic data?
In this case we would not want to stratify because we don’t care about demo and stratifying would introduce potential modeling error, no?
OK, I admit I didn’t MP’s blog until two weeks before the election. I withdraw my question on stratified polling, as I’m sure that’s a time tested practice. I’ll look it up in MP’s archives to learn more.
But I would comment that when we don’t care about demo data, and simply want to know how the full population voted, it may be possible to simplify polling procedures. An audit of an election has different priorities than standard polling, I imagine, so I wonder if MP might comment on how an “audit” poll might differ from regular polls.
Thanks!
Alex, actually, with a survey of more than 16,000 persons, I don’t think stratification would prove to be that big of a deal. No way to tell though until after you’ve drawn your initial sample and see how things fall. If you find that some key demographics or geos are underrepresented in the sample (and therefore the sample is not “characteristic” of the population you are trying to survey), then you could simply draw more samples until you get the right amount (I think this is best, but costly), or you could apply weights and corrections. Either way, this is more important for smaller sample sizes. That’s what I think anyway…
I’m not a methodologist, but I’d be interested to hear from David why he would want 0.05% accuracy. I think I must be misunderstanding something, because that seems to be unnecessary precison.
If I sample enough people such that I can be 99% sure that the two margins of error (confidence intervals) I’m interested in –Kerry’s and Bush’s percentages, in this instance– don’t overlap, that seems like it would suffice. It would probably be good enough to be merely 95% confident–that’s good enough to make a lot of money winning bar bets, which is the practical standard for confidence.
After reading a bit in my old grad-school text (Hays) I can’t think of any reason why it would take more than a few thousand samples to shrink the confidence intervals to the point where they positively do not overlap (i.e., where we can be 95% or 99% confident that the person who seems to have got more really did get more).
Examining the “Unexplained Exit Poll Discrepancy”
A day after Edison/Mitofsky released their much anticipated report on the 2004 Presidential Election exit polls, the University of Pennsylvania issued a press release announcing that an “expert” on the presidential election exit poll errors has access …
FAQ: Questions about Exit Polls
I wrote quite a bit about exit polls after the 2004 elections, and have learned quite a bit in the process. However for those looking for general information about how exit polls are conducted, my Election Day summary of “what…
I know this is way old, but, first off, about the Utah Colleges Exit Poll – I’m one of the supervisors of the Exit Poll, the reason why we have a high response rate is because of the connection between locals and the universities – lots of Utahns stay in Utah to go to school, so they’re more likely to respond. Of course, the huge sample size helps with accuracy.
Now, as far as Mairead’s comment, the main reason you’d want .05% accuracy is that if you’re in a 95% ME, there’s still a chance that the true value is outside of your range, if Bush and Kerry have 95% CI where the low and high of each are close – you could still be wrong. A 99.95% CI would be near perfect.
The fact of the matter is, people lie all the time on survey’s – because they hate pollsters, and they try and mess stuff up. Or their concerned about a violation of privacy, or any number of other things. So, while Exit Polls are very accurate – remember, it’s a lot easier to produce fake Exit Poll results than a fake election – and if the pollsters, or the people who hired them, have an agenda, you can easily design an exit poll that will skew results one way or another without making it apparent. A cluster sample of polling places could very easily be un-random, favoring pro-Kerry regions. In the Utah Colleges Exit Poll we intentionally include extra high-democrat polling places to make sure we have enough information about the democrats – if we didn’t, we might over represent the Republican vote.
Just some thoughts.
Oh, and our interviews are trained like crap in the Utah Colleges Exit Poll. They don’t know what they’re doing. Luckily, they just hand off a survey sheet and the person fills it in on their own.