« October 31, 2004 - November 6, 2004 | Main | November 14, 2004 - November 20, 2004 »

November 12, 2004

Pew on "Moral Values"

The Pew Research Center released a new study yesterday, a follow-up interview with 1203 voters who were originally surveyed in October that sheds interesting new light on the “moral values” controversy.

Pew conducted an experiment with the question that asked, “What one issue mattered most to you in deciding how you voted for president?” For a random half sample, they provided the same list of fixed choices as the national exit poll: Moral values, Iraq, economy/jobs, terrorism, health care, education and taxes. They asked respondents on the other half sample to answer in their own words, and did not provide a list of choices.

The survey’s findings parallel exit poll results showing that moral values is a top-tier issue for voters. But its relative importance depends greatly on how the question is framed. The post-election survey finds that, when moral values is pitted against issues like Iraq and terrorism, a plurality (27%) cites moral values as most important to their vote. But when a separate group of voters was asked to name - in their own words - the most important factor in their vote, significantly fewer (14%) mentioned moral values.  Regardless of how the question is asked, the survey shows that moral values is the most frequently cited issue for Bush voters, but is seldom mentioned by Kerry voters.

In addition, those who cite moral values as a major factor offer varying interpretations of the concept. More than four-in-ten (44%) of those who chose moral values as the most important factor in their vote from the list of issues say the term relates to specific concerns over social issues, such as abortion and gay marriage. However, others did not cite specific policy issues, and instead pointed to factors like the candidates' personal qualities or made general allusions to religion and values.

The consistency across methodologies is arguably more important the differences: Whether asked as an open-ended or closed-ended question (a) no single issue dominates and (b) Bush voters mention issues related to “moral values” most often, while Kerry voters rarely mention such issues.

If you care about this controversy, their report is worth reading in full.

Posted by Mark Blumenthal on November 12, 2004 at 10:44 AM in Exit Polls, Measurement Issues, The 2004 Race | Permalink | Comments (6)

Lessons: Likely Voter Models & Bias

On Wednesday morning after the election, still a bit groggy, I cobbled together a quick table of the final surveys released by the major national polling organizations. I calculated an overall average (showing a Bush margin of 1.7%), compared that to Bush’s actual 3.0% margin and concluded, “the traditional likely voter models performed reasonably well.”

A few hours later I received an email from a loyal reader, Prof. X: “I'd take issue with your view that the average of the polls is close,” he wrote. “Mostly, I think this is dead wrong.”

When Prof. X tells me I’m “dead wrong” I tend to pay close attention because he is way ahead of me in statistical training and smarts generally. So I decided to take a second look. A slightly updated version of the table appears below.


A few minor notes: This table includes the final survey for each organization rather than “projections” that allocate the undecideds. In one case (Zogby) I used the next to last release of data that still had undecideds included. Also, I have reported only one set of numbers for ABC/Washington Post, whose final numbers were identical. Finally, I have intentionally omitted two surveys by Harris and the Economist that were drawn from Internet panels. I will take up the lessons from projections and Internet polls separately.

One of Prof. X’s complaints was that by including three surveys fielded a full week before the election (by ICR, Time and the LA Times), I “artificially made [the average] look better than it is.” Fair enough. This time I included a separate average based only on surveys done in the final week that brings the average Bush margin down slightly to 1.5%, roughly half that of Bush’s latest margin of the popular vote (2.9%).

I have to admit that my conclusion that the margin was “reasonably close” would have been the same had I calculated it at 1.5%.  I was thinking the way any pollster does when they compare their final poll to reality. A three point Bush win (51% to 48%) is within +/-3-4% sampling error of a poll giving Bush a one-and-a-half-point lead (49% to 47.5%).

However, that margin of error applies to only to one survey at a time.  As Prof. X pointed out, the same logic does not apply to an average of 15 or more polls. In fact, most of the surveys done in the final week (11 of 15) had a “bias” toward Kerry – they showed Kerry ahead or had the margin closer than three points. That result cannot be explained by random variation: My application of the binomial distribution puts the probability of that happening by chance alone (assuming that undecideds broke evenly) at roughly 6%.

So why did so many national polls show the race closer than it turned out to be?

1) Undecideds “broke” two-to-one towards the incumbent George Bush. I’ll concede that the incumbent rule (under which undecided voters on the final survey typically break to challengers) did not hold in this year’s presidential race, but I’m dubious of a break towards Bush. In the national exit poll, those who made their final vote choice in the last four days went for Kerry 53% to 44%, while those deciding earlier went for Bush (52% to 47%). The same pattern held in Florida and Ohio. The average of the national tracking surveys also showed no trend to Bush over the final weekend; if anything, they had Kerry running slightly closer.

2) Some likely voter models worked better than others - When I presented the details on likely voter models, I noticed that the pollsters that used a variant of the Gallup likely voter model showed Bush doing consistently better than other surveys. That difference now looks prescient. The following table shows the results of those using the Gallup likely voter model either in the final week (Gallup, Pew, Newsweek) or in the final two weeks (adds Time and the LA Times). In both cases, the Gallup-model showed a Bush margin closer to the actual result (3.2%+) than the average of the other surveys (0.9%). There were three surveys in the “other” category that correctly forecast Bush’s final three-point margin (notably, Pew, TIPP, ICR), but the other 10 showed Kerry doing slightly better.


3) National pollsters manipulated their results – consciously or unconsciously – to show a closer result - One cynical reader wrote just before the election to suggest what he called “price fixing” at the national level: He argued that some national surveys were intentionally “’managing’ (or massaging) the data to have it come out Bush +1 or Bush +2.”

Crazy? Perhaps, but as the same reader correctly anticipated, the small Kerry bias in the national surveys did not appear at the state level. RealClearPolitics provided averages of final polls in 18 battleground states. Kerry did better than the RCP average in nine states while Bush did better in eight states – about what we would expect by chance alone.

4) Respondents were more supportive of Kerry than non-respondents - This possibility would be easier to dismiss had not the exit polls shown a similar deviation in Kerry’s favor. The exit pollsters are speculating that non-response bias may be the reason.

Which explanation is most convincing? The truth is probably some combination of #2-4 above, and each topic is worthy of further investigation.

I am wary of picking individual “winners” among the national polls, as any one survey can hit or miss the precise final result by chance alone. Having said that, the most obvious lesson in all of this is that the Gallup likely voter model as employed collectively by Gallup, Pew, Newsweek and others came closer to predicting the outcome in the final week than the average of the other surveys. The notable exceptions, TIPP and Battleground, are also worthy of further discussion, since both weighted their surveys by party ID. More to come on that…

(My apologies to those who viewed an intially garbled version of this post between 8:30 and 9:00 a.m. EST.  My weblog host, Typepad, has chosen to implement new "features" without much of a beta test.  Very frustrating)

Posted by Mark Blumenthal on November 12, 2004 at 08:24 AM in Divergent Polls, Likely Voters | Permalink | Comments (6)

November 09, 2004

Blaming the Bloggers - epilogue

Yesterday's "Real Time" column by the Wall Street Journal's Hanrahan and Fry made essentially the same point I did about the exit poll/blogger controversy.

My point, for those who may have missed it, was less about defending "the likes of Wonkette" (whatever that means) than acknowledging that the existance of bloggers makes the rapid proliferation of highly sought after information (or raw data or whatever you want to call it) inevitable and automatic.  Here is how Hanrahan and Fry put it:

The reaction to this whole controversy is so 1997 -- a predictable volley of off-target shots by chin-waggers who either haven't figured out that the world's changed or refuse to admit that it has...

What's changed is that once you had to be a media or political person of a certain minimum altitude to be let in on the secret; now, all you need is a Net connection and rudimentary typing skills. Those who used to have the clubhouse all to themselves have found the door wide open to anyone and everyone. And some of them don't like it.

Which, come to think of it, really is so 1997: Information formerly reserved for a cadre of insiders now spreads more quickly than those insiders can control it. In other words, the middlemen have been eliminated. (If you spent 100 large at business school, you call that "disintermediation.") Just took a bit longer to hit the Election Day carnival, that's all.

Well said.  I am told the Wall Street Journal Online is free to non-subscribers this week, so read it all

P.S.  Say what you will about Ana Marie Cox, in addition to the leaked exit polls, she also posted (at 5:40 p.m. Election Day) the following leaked disclaimer that the National Election Pool officials had shared with the network honchos: “problems with exits in the following states...could be tipping numbers toward kerry: MN, NH, VT, PA, VA, CT, DE.” Not that it mattered. This was the same disclaimer that the editors of the Post and the Times either did not hear or discounted like everyone else.

Posted by Mark Blumenthal on November 9, 2004 at 02:49 PM in Exit Polls | Permalink | Comments (11)

Vote Fraud?

I am hoping to spend the next week or so looking at what the elections have told us about the various issues we discussed here in the weeks before the election: likely voter models, weighting by party, the incumbent rule, etc. But first, one last thought about exit polls.

Many readers have emailed about various summaries of exit poll results collected at different times on election night. Many are looking for evidence of fraud (or looking to debunk claims of fraud) in the apparent discrepancies between the vote count and the “early” results of the exit polls.

Exit polls are certainly a powerful tool in uncovering vote fraud, when it exists. If, as some allege, a particular type of voting equipment was used to perpetuate a wide scale deception last week, the raw exit poll data could help reveal it. I’m dubious about these claims, to say the least, but that’s my opinion. Those who disagree are right to press the exit pollsters for more answers.

Unfortunately, the various exit poll results that were captured online on election night are not much help. First, a lot of the speculation classifies the type of voting equipment used at the state level. Most states use a combination of different types of counting equipment. In Illinois, for example, equipment varies by county (as this web site shows). Some variation may occur within counties or even individual precincts. My precinct in Washington DC, for example, allowed voters to choose between an optical scan ballot and an experimental touch screen voting machine.

Second, regarding the exit poll results captured at various times of the evening: We may know the time they appeared on a given website, but we do not know how old those tabulations were when they were posted, and more important, we do not know the extent to which any given sample was corrected to conform with the actual vote.

That last point is important and not well understood. Remember two things: First, the weighting of exit polls to match actual results is not new, but a standard procedure used since the early days of exit polls. Second, the weighting to actual returns does not occur all at once but continuously, precinct by precinct, over the course of election night. The exit pollsters weight their sample to match incoming actual results for each sampled precinct as actual returns become available. Thus, the exit poll results get continuously updated in what bloggers might call “real time.” Some of the online postings may reflect that updating; some may not. We have no way of knowing. There is also one more step: The sampled precincts are still just a sample, so even when all the sampled precincts have been weighted to the actual result, sampling error may cause the survey to differ from the statewide result. At that point, near the end of vote counting, the exit pollsters will apply another overall weight so that the vote on the survey matches the actual statewide result.

Although the data now in the public domain is not much help, the raw data puts the exit pollster in a strong position to evaluate some of the speculation about vote fraud. If, for example, someone tampered with tabulations from touch screen voting machines that lacked a paper trail, then an analysis of the poll data should show a greater discrepancy in precincts with such machines.

A New York Times article that appeared last Friday implies that such an analysis has already occurred. The National Election Pool (NEP) officials that conducted the exit polls wrote a report to debrief their clients on the apparent mishap. The Times article said their report “debunked” the theory of a fraudulent vote count, but did not elaborate. As far as I know, that report has not been made public.

Skeptics are certainly right to want to see the data behind that conclusion. As should be obvious, the raw data is under the control of the organizations -- ABC, CBS, CNN, FOX, NBC and the Associated Press – that collectively own it. In past years they have deposited the raw data at the archives of the Roper Center, where it is available for analysis to the general public. Mr. Kaus is right: It would be relatively easy for a major network or newspaper to use the raw exit poll data to debunk – or at least explore – the claims of voter fraud. I’m all for it.

One last thought: I said I am dubious that this data will show evidence of vote tampering. Why? No pollster ever wants to be wrong, to have their numbers called into question. Yet, fair or not, that is exactly what people are saying about last week’s exit polls.

The NEP officials who ran this year’s exit polling, Warren Mitofsky and Joe Lenski, are people I greatly respect. They are not partisans, but extraordinarily skilled survey methodologists.  Mitofsky, along with a small team of colleagues at CBS, helped invent the exit poll almost 40 years ago, along with many other methods and models still in use by survey researchers today. Although no one is infallible, Mitofsky is deservedly a legend in the field of survey research. Yet despite that stellar standing, last week’s perceived glitch still threatens his reputation and the continuing livelihood of his collegues. 

So apply a bit of common sense: If Mitofsky has evidence that his exit poll was right but the vote tally was wrong, do you believe for a minute that he would suppress it? I certainly do not.

Posted by Mark Blumenthal on November 9, 2004 at 12:19 PM in Exit Polls | Permalink | Comments (19)

November 07, 2004

Blaming the Bloggers

One thing that has bothered me about the coverage of the early exit poll snafu on Tuesday is the notion that bloggers were the problem. "Bloggers Said to Blame for Bad Poll Info," read the headline over the AP story in the New York Times, consistently echoing the line from those close to the exit poll process:

  • Joe Lenski, whose Edison Research conducted the NEP exit poll, said: "The basic issue here is the leaking of this information without any sophisticated understanding or analysis."
  • CBS News Polling Director Kathy Frankovic said: "I think people believed [the leaked polls], and it's particularly the case with Internet bloggers."
  • Former CBS News Executive Political Director Martin Plissner wrote: "The problem is not that the exit polls were wrong...the problem was that in the age of the Internet the exit polls were being seen by thousands of people who didn't know how to read them."

The New York Sun headline said it all: "Bloggers Botch Election Call; Networks Cautious, Steady."

It is certainly true that the networks were cautious, steady and ultimately flawless in making their formal projections. But the idea that only bloggers -- as opposed to "sophisticated" journalists -- leapt to wrong conclusions is nonsense.

Hopefully, as a blogger who not only declined to post leaked exit poll numbers but also explicitly forbid readers from sharing leaked data on this site (see note below), I have some credibility on this issue

I spent much of late Tuesday afternoon watching CNN. Here's what I saw. In the 5:00 hour, Lou Dobbs, Paul Begala and Tucker Carlson had this exchange:

BEGALA: If the early projections look like they're true, turnout will be way up this year...I do think -- I am, of course, a Kerry partisan, but I do think it's probably good news for John Kerry. The higher the turnout goes, probably the better it is for the challenger. As one Kerry aide said to me this afternoon, he said, you know, you don't stand in line for 45 minutes to vote for more of the same.

DOBBS: Do you agree with that, Tucker?

CARLSON: Sadly, I do agree with that. I can tell you, Democrats agree with that.

CARLSON: I listened to two well-known Democrats -- I won't tell you their names -- literally involved in a conversation of who was going to be Kerry's White House chief of staff. They are picking the curtains in the White House, which is reason enough in my view to hope they don't win.

The gloating has begun already. But, yes, they definitely think it's good for them. And I think, sadly, they're probably right.

Translation: Carlson and Begala were both convinced by the partial exit polls that heavy turnout meant "good news" for Kerry. In a commercial break, Carlson listened to Begala and James Carville speculating on who Kerry would pick as chief of staff. Presumably, CNN had briefed them on the partial exit poll numbers, and everyone was convinced Kerry would win. Remarkably, on the air, Carlson all but conceded a Bush defeat.

Oh, but Carlson and Begala are not real journalists, you say? Then consider this exchange a few minutes later between Dobbs, Karen Tumulty of Time Magazine and Roger Simon of U.S. News & World Report (the transcript has two parts, one and two). Keep in mind that at the time this segment aired, the national exit poll showed Kerry leading 51% to 48%:

DOBBS: And, Karen, the long lines that we're seeing, conventional wisdom has been that a heavy turnout would favor Senator Kerry. Anything that you want to add to that conventional wisdom?

KAREN TUMULTY, NATIONAL POLITICAL CORRESPONDENT, "TIME": You know, I haven't seen anything that would shake that conventional wisdom today...

ROGER SIMON, POLITICAL EDITOR, "U.S. NEWS & WORLD REPORT": I agree with Karen. The motivation is that, if you want change, you get out and vote. If you're satisfied with the status quo, you're more likely to stay home, and that probably favors John Kerry

TUMULTY: Yes, I am feeling more so than I was a day or two ago confident that we, in fact, may know the winner by the time we go to bed tonight, assuming we go to bed in the wee hours, that is, and, you know, I think that it does appear that this turnout could perhaps create a wave.

DOBBS: Your thoughts, Roger?

ROGER SIMON, U.S. NEWS & WORLD REPORT: The same. I think we'll have a winner. I think he will win 51:48:1. I just don't know which guy will get that, but I think...


SIMON: But I think we will have a clear winner. That win will be magnified in the electoral college. It won't go to the House of Representatives, and we'll know who's president tomorrow morning, and we can all either be angry or be sad or be happy, but breathe a sigh of relief that we know.

Translation: Yesterday we thought things would be close, but now we know it will be blowout.  In fact, we think Kerry is going to win 51% to 48%, with an even bigger electoral college margin because he is leading in all the battleground states. We're even using the actual numbers on the air while still claming we don't know who will win! Aren't we clever? What a hoot!

Some say it was just the talking heads on the networks that fell into this trap. We now know the editors at the nation's most prestigious newspapers -- relying not on leaks but on data received via paid subscription -- acted on the belief Kerry would win. The managing editor of the Washington Post said his paper "had to scramble to make last-minute changes to an article analyzing why voters voted the way they did that was based in part on the poll." An article by Jim Rutenberg said the New York Times also had to pull "an analytical piece about the vote based in part on the Election Day survey" from late editions of the paper.

So my advice to decision makers at the networks is to stop blaming the bloggers. If you put exit poll data into the hands of hundreds of working journalists, it will absolutely, positively proliferate to millions of readers via weblogs. The genie is out of that bottle for good. Blaming bloggers for spreading that information is like blaming fish for swimming.

If the public release of partial exit poll data is too "dangerous" for "amateurs," then think about keeping it under wraps until the polls close. That means sharing it with a very small number of people in a central location and no ability to communicate with the outside world. Impossible? Why not? Somehow, the Academy Awards manages to keep its tabulations secret each year. I am not sure why exit poll data needs to be different, except that powerful people feel entitled to an early peek.

And if true secrecy really is as unworkable as everyone connected with the exit polls keeps telling me it is, then consider making the whole process a lot more transparent. Consider this suggestion from Political Scientisteconomist/blogger Daniel Drezner:

I have a humble request for the nets -- show us how the sausage is made. In other words, instead of hiding the data from the exit polls from us, explain as the returns come in what the polls say and compare and contrast them to the incoming returns.

Crazy? Perhaps, but at least this way the public will have an explicit warning label from the source to go with that oh, so dangerous data.


A note to readers: On Tuesday, I asked you to refrain from posting leaked exit polls on this site. I did so, not because I imagined I could slow their dissemination, but because I thought it hypocritical to criticize the discussion of leaked exit polls while simultaneously facilitating it. When I posted, I unthinkingly left the "trackback" feature on, which allows other blogs to post remote comments and have those comments appear automatically at the bottom of my post. By 6:00 p.m. EST, 36 blogs had linked to the site, and virtually all were posting or commenting on the leaked exit polls. A few even included leaked numbers in the summary blurbs that appeared here. A lesson learned: The very structure of the blogosphere facilitates the dissemination of information. Resistance, as they say on Star Trek, is futile.


Correction:  Daniel Drezner reminds me that he is more Political Scientist than  Economist.  My goof.   

Update:  Jack Shafer's column Friday night covered a lot of the same turf as this post.  So did this piece by the Wall Street Journal's Hanrahan and Fry.  Both are worth reading in full.

Posted by Mark Blumenthal on November 7, 2004 at 07:23 PM in Exit Polls | Permalink | Comments (8)


Saturday's New York Times had three articles on the other big exit poll issue this week: The question that showed 22% of voters choosing "moral values" as the issue they were most concerned about.

In an op-ed piece, ABC News Polling Director Gary Langer presented a methodological critique:

The exit poll...asked voters what was the most important issue in their decision: taxes, education, Iraq, terrorism, economy/jobs, moral values or health care. Six of these are concrete, specific issues. The seventh, moral values, is not, and its presence on the list produced a misleading result.

How do we know? Pre-election polls consistently found that voters were most concerned about three issues: Iraq, the economy and terrorism. When telephone surveys asked an open-ended issues question (impossible on an exit poll), answers that could sensibly be categorized as moral values were in the low single digits. In the exit poll, they drew 22 percent.

On the same page, David Brooks' column noted Andrew Kohut's analysis of the exit polls:

As Andrew Kohut of the Pew Research Center points out, there was no disproportionate surge in the evangelical vote this year. Evangelicals made up the same share of the electorate this year as they did in 2000. There was no increase in the percentage of voters who are pro-life. Sixteen percent of voters said abortions should be illegal in all circumstances. There was no increase in the percentage of voters who say they pray daily.

In the news section, Jim Rutenberg covered the controversy, including a rejoinder from Republican pollster Bill McInturff, who called the critiques "garbage:"

"The people who picked moral values as an issue know what that means," he said. "It's a code word in surveys for a cluster of issues like gay marriage and abortion."

Mr. McInturff said that if "moral values" was really a "catchall" with a confused meaning, then more Democrats would have picked it. Of the 22 percent who chose "moral values," 80 percent were Bush supporters, 20 percent were Kerry supporters. "It's self-selected by people for whom these issues are very important for their votes," he said, adding that the margin by which Mr. Bush carried these voters arguably made the difference in the election.

Who is right? I agree with Langer and Kohut that the use of a closed-ended question on the exit poll exaggerated the percentage of voters who would have volunteered "moral values" as a response. Had the exit polls been able to ask an open-ended question, the results would have been comparable to those obtained from telephone surveys during the campaign.

However, Democrats are right to conclude they "need to do a better job connecting with cultural traditionalists," as Rutenberg's piece put it. Consider that in a survey conducted in mid October by the Pew Research Center, 63% of registered voters considered "moral values" very important in deciding their vote. That percentage was less than for issues like the economy, jobs, terrorism, Iraq, education and health care (which ranged from 73% to 78% very important), but about the same as Social Security (65%) and more than issues like taxes (59%), the budget deficit (57%) and the environment (53%). And in the NBC/Wall Street Journal poll conducted in mid-October, registered voters thought George Bush would do a better job on "issues related to moral values" than John Kerry by an 18 point margin (47% to 29%).

Posted by Mark Blumenthal on November 7, 2004 at 07:45 AM in Exit Polls, Measurement Issues, The 2004 Race | Permalink | Comments (6)