October 28, 2004
Likely Voters VI: Still More on Gallup
Of the two complaints about the Gallup likely voter (LV) model I covered in the last post, the first (that the selection procedure does not perfectly predict individual turnout) applies equally to all LV models. The second (that it is too volatile) appears mostly directed at Gallup, but many of the similarly structured surveys (Pew, Newsweek, Time) have been more volatile than surveys that either weight by party (Zogby, TIPP, Rasmussen, ABC/Washington Post tracking, sometimes WSJ/NBC) or stratify by turnout (FOX/Opinion Dynamics, Democracy Corps).
A third complaint is mostly about Gallup: A chorus of voices on the Kerry-wing of the blogosphere and elsewhere - especially Ruy Teixeira (DonkeyRising), Steve Soto (TheLeftCoaster) and Chris Bowers (MyDD).
First, are they right? I put the numbers posted on RealClearPolitics into my spreadsheet and found:
- This week, Gallup has Bush ahead by five points (51% to 46%), the other surveys have Bush ahead by an average of two points (48% to 46%).
- This week, Gallup released surveys in six states. The averages across all six states are similar to the national result Gallup has Bush ahead by an average of 4 points (50% to 46%), others have Bush up by an average of one point (48% to 47%). Gallup gives Bush a bigger lead in 4 of 6 states.
- Since the Democratic convention, Gallup has released 10 national polls. Although the differences were sometimes small, 7 of 10 showed a bigger margin for Bush than the average of other likely voter surveys conducted over the same period. If you average the averages: Gallup had Bush ahead by an average of five points (50% to 45%), while others had Bush head by two points (47% to 45%).
- If the pattern seems weak, consider this: In 11 of 16 cases cited above, Bush did better on Gallup surveys, something that should have been a 50/50 proposition each time. The probability of flipping a coin 16 times and having it come up heads 11 times is 6.7%.
I want to be clear. I do not believe that anyone at Gallup, CNN or USAToday has intentionally skewed its numbers. But they have been showing Bush doing a bit better than other surveys. This raises two questions: (1) Why are they different and (2) who is right?
Steve Soto has been arguing for months that Gallup's samples are biased toward Republican identifiers. He obtained and posted party identification results from Gallup for nearly every survey it releases, and has argued that Gallup's party mix seems implausible in comparison to party registration statistics or past exit poll results. Ruy Teixeira and Alan Abramowitz have made similar arguments on Donkey Rising. Soto and Teixeira have also recently written about the minority representation in Gallup's most recent sample: 8% of likely voters were black, compared with 10% of voters on the 2000 exit polls; 15% of Gallup's likely voter sample was non-white compared to 19% of voters on the 2000 exit polls. They show similar differences for income, as well.
I do not want to revisit the debate about weighting by party identification. I have misgivings (you can read all about those starting here). However, in this case I agree with Teixeira that the demographic differences are less the disease than a symptom. They are telling us to look carefully at the effect of the likely voter model.
Let me suggest another discrepancy. I called Gallup seeking answers to a few questions I had about how first time voters could possibly qualify as likely voters. Since they say a likely voter must score at least a 6 out of 7 on their index, and since three of the items require some past voting, it would be impossible for a first time voter to qualify. I had asked this question via email about a month ago, and was told that Gallup gives 18-21 year olds an extra point. But the extra point would still leave a first time voter with a maximum of five points.
The answer, I learned from Gallup's Jeff Jones yesterday, is that younger voters get more than one bonus point. Apparently, Gallup gives 18 and 19 year olds up to three extra points, depending on how they score on the other likelihood questions. They give 20 and 21 year olds up to two points (since they could have voted in 2002 and answered that they "always" or "almost always" voted in prior elections).
Still with me?
Here's the bottom line. On a self-reported question, 6% of those who qualified as likely voters said they will cast their first presidential vote in the 2004 election. Among all registered voters, 12% say they are first time voters. Though a very small subgroup (roughly 72 weighted interviews) first-time likely voters support John Kerry by a 56% to 42% margin, while past voters prefer Bush 52% to 45%.
The difference in the Gallup survey looks to me to be right on the edge of statistical significance. However, it is consistent with preference for Kerry among first time voters on two other recent surveys: 57% to 36% for Newsweek and 54% to 43% for ABC. Also on the 2000 exit polls, Al Gore won first time voters by a 52% to 43% margin. The real distinction for Gallup was the percentage of likely voters that qualified.
Again, self-reported first-time voters were:
- 6% of likely voters on Gallup's most recent survey
- 9% of likely voters on Newsweek's recent survey
- 10% of likely voters on the recent ABC survey
- 9% of voters in 2000, according to the national exit poll
Jones points out that even if they had doubled the number of first time voters in the sample, it would have cut Bush's overall margin by only a single percentage point. True. But the lower number raises a larger question. Is the Gallup model simply screening out too many voters who do not typically vote in presidential elections? All mechanical issues aside, the demographic differences and the higher than average support for Bush (given the consistent finding that non-2000 voters tend to prefer Kerry) suggest that Gallup is effectively modeling a lower turnout than the other surveys.
Every measure of intent to vote and interest in the election is significantly higher this year on every survey I have seen. According to Jones, 84% of adults now say their probability of voting rates a 10 on Gallup's 1-10 scale, 16 percentage points higher than this time four years ago (68%). Not all of those adults will vote, to be sure, but the finding certainly suggests a higher than usual turnout. Shouldn't categories like first-time voters and self-described non-voters from 2000 be a bit higher than usual rather than lower?
Perhaps Gallup agrees. According to Jones, they will raise their cutoff for likely voters on their last survey coming up this weekend from 55% of adults to 60%. Jones and others with access to the Gallup data tell me such a change would not have altered the results much on the last few Gallup surveys. Perhaps. But if that's the case, why change now?
Related Entries - Likely Voters
I wonder how many of those first-time voters are in battleground states? In about 40 states, the first time voters are largely irrelevant, because either Bush or Kerry (depending on the state) are far ahead.
In OK, for example, Bush apparently is ahead by 30 points. First-time voters there, like their counterparts in CA, NY, MA, RI, and VT, will be doing their civic duty. They won't be influencing the election.
I'd be very interesting in the likelihood of first-time voters turning out in PA, MI, NJ, OH, and a few other states.
The issue is trickier than it might appear: in PA, Democrats have registered more new voters in 2004 than Republicans, although not by a wide margin. However, since the last presidential election in 2000, the Republicans have registered substantially more people than the Dems.
I have a question: how much is Kerry's big lead in huge states (NY, CA, IL) skewing upward his national poll averages?
Posted by: Steve | Oct 28, 2004 9:00:22 PM
You say "According to Jones, they will raise their cutoff for likely voters on their last survey coming up this weekend from 55% of adults to 60%. Jones and others with access to the Gallup data tell me such a change would not have altered the results much on the last few Gallup surveys. . Perhaps. But if that's the case, why change now?"
Isn't a logical reason for the methodolgical change, especially if it probably won't change the results, is in response to the constant harping of democrat leaning blogs and moveon.org?
Posted by: Andrew Sacher | Oct 28, 2004 9:24:28 PM
I really wish they wouldn't change their methodology until the election is over because now I'll be comparing apples to oranges. I won't know whether a Gallup change is due to a real surge or simply because of their methodology change. If they really want to do this, they should publish both the 55% and the 60% results.
But, well, Gallup hasn't asked me for my opinion.
Posted by: carrie | Oct 29, 2004 2:35:09 PM
Great discussions, Mark. I want to raise a point about the methodology of choosing likely voters. I think it is totally incorrect, statistically, to arrive at a score for each voter, rank the results and simply apply a cut off, as Gallup does.
Any poll is attempting to use a sample to describe the population from which it is drawn. This cutoff method clearly and logically fails to achieve that. Why? Because it is extrememly doubtful that the turnout among 7's will be 100%, or that the turnout among, say 5's (who won't make the cut) will be 0%. Rather, each score should be given a "likelyhood" probability, and ALL registered voters should then be included, weighted by the probability of them voting. The probability could be scaled to reach a target turnout, say 55%, 60% etc.
Reasonable probabilities for each score could surely be obtained from past data.
What the current Gallup methodology does is totally, 100%, eliminate many geniuine first time voters who may not have had a strong interest in prior elections. And one cannot simply assume that the cutoff is unbiased. The evidence is overwhelming that this cutoff method introduces a significant candidate bias.
A cutoff may be the best "guess" for the poll sample, but it most certainly does not describe the voting population. In this respect it is seriously flawed.
Mark, what are your thoughts on weighting by probability, rather than applying an arbitrary cutoff? I really believe this achieve far more accurate results.
Posted by: Antony | Oct 30, 2004 2:24:50 PM
Haha! I should have carried on to "Likely Voters VII" where you cover the topic in discussing CBS's method, which is pretty much as I described.
Posted by: Antony | Oct 30, 2004 2:47:22 PM
The comments to this entry are closed.