Likely Voters VI: Still More on Gallup

Of the two complaints about the Gallup likely voter (LV) model I covered in the last post, the first (that the selection procedure does not perfectly predict individual turnout) applies equally to all LV models. The second (that it is too volatile) appears mostly directed at Gallup, but many of the similarly structured surveys (Pew, Newsweek, Time) have been more volatile than surveys that either weight by party (Zogby, TIPP, Rasmussen, ABC/Washington Post tracking, sometimes WSJ/NBC) or stratify by turnout (FOX/Opinion Dynamics, Democracy Corps).

A third complaint is mostly about Gallup: A chorus of voices on the Kerry-wing of the blogosphere and elsewhere – especially Ruy Teixeira (DonkeyRising), Steve Soto (TheLeftCoaster) and Chris Bowers (MyDD).

First, are they right? I put the numbers posted on RealClearPolitics into my spreadsheet and found:

• This week, Gallup has Bush ahead by five points (51% to 46%), the other surveys have Bush ahead by an average of two points (48% to 46%).
• This week, Gallup released surveys in six states. The averages across all six states are similar to the national result Gallup has Bush ahead by an average of 4 points (50% to 46%), others have Bush up by an average of one point (48% to 47%). Gallup gives Bush a bigger lead in 4 of 6 states.

• Since the Democratic convention, Gallup has released 10 national polls. Although the differences were sometimes small, 7 of 10 showed a bigger margin for Bush than the average of other likely voter surveys conducted over the same period. If you average the averages: Gallup had Bush ahead by an average of five points (50% to 45%), while others had Bush head by two points (47% to 45%).

• If the pattern seems weak, consider this: In 11 of 16 cases cited above, Bush did better on Gallup surveys, something that should have been a 50/50 proposition each time. The probability of flipping a coin 16 times and having it come up heads 11 times is 6.7%.

I want to be clear. I do not believe that anyone at Gallup, CNN or USAToday has intentionally skewed its numbers. But they have been showing Bush doing a bit better than other surveys. This raises two questions: (1) Why are they different and (2) who is right?

Steve Soto has been arguing for months that Gallup’s samples are biased toward Republican identifiers. He obtained and posted party identification results from Gallup for nearly every survey it releases, and has argued that Gallup’s party mix seems implausible in comparison to party registration statistics or past exit poll results. Ruy Teixeira and Alan Abramowitz have made similar arguments on Donkey Rising. Soto and Teixeira have also recently written about the minority representation in Gallup’s most recent sample: 8% of likely voters were black, compared with 10% of voters on the 2000 exit polls; 15% of Gallup’s likely voter sample was non-white compared to 19% of voters on the 2000 exit polls. They show similar differences for income, as well.

I do not want to revisit the debate about weighting by party identification. I have misgivings (you can read all about those starting here). However, in this case I agree with Teixeira that the demographic differences are less the disease than a symptom. They are telling us to look carefully at the effect of the likely voter model.

Let me suggest another discrepancy. I called Gallup seeking answers to a few questions I had about how first time voters could possibly qualify as likely voters. Since they say a likely voter must score at least a 6 out of 7 on their index, and since three of the items require some past voting, it would be impossible for a first time voter to qualify. I had asked this question via email about a month ago, and was told that Gallup gives 18-21 year olds an extra point. But the extra point would still leave a first time voter with a maximum of five points.

The answer, I learned from Gallup’s Jeff Jones yesterday, is that younger voters get more than one bonus point. Apparently, Gallup gives 18 and 19 year olds up to three extra points, depending on how they score on the other likelihood questions. They give 20 and 21 year olds up to two points (since they could have voted in 2002 and answered that they “always” or “almost always” voted in prior elections).

Still with me?

Here’s the bottom line. On a self-reported question, 6% of those who qualified as likely voters said they will cast their first presidential vote in the 2004 election. Among all registered voters, 12% say they are first time voters. Though a very small subgroup (roughly 72 weighted interviews) first-time likely voters support John Kerry by a 56% to 42% margin, while past voters prefer Bush 52% to 45%.

The difference in the Gallup survey looks to me to be right on the edge of statistical significance. However, it is consistent with preference for Kerry among first time voters on two other recent surveys: 57% to 36% for Newsweek and 54% to 43% for ABC. Also on the 2000 exit polls, Al Gore won first time voters by a 52% to 43% margin. The real distinction for Gallup was the percentage of likely voters that qualified.

Again, self-reported first-time voters were:

• 6% of likely voters on Gallup’s most recent survey
• 9% of likely voters on Newsweek’s recent survey
• 10% of likely voters on the recent ABC survey
• 9% of voters in 2000, according to the national exit poll

Jones points out that even if they had doubled the number of first time voters in the sample, it would have cut Bush’s overall margin by only a single percentage point. True. But the lower number raises a larger question. Is the Gallup model simply screening out too many voters who do not typically vote in presidential elections? All mechanical issues aside, the demographic differences and the higher than average support for Bush (given the consistent finding that non-2000 voters tend to prefer Kerry) suggest that Gallup is effectively modeling a lower turnout than the other surveys.

Every measure of intent to vote and interest in the election is significantly higher this year on every survey I have seen. According to Jones, 84% of adults now say their probability of voting rates a 10 on Gallup’s 1-10 scale, 16 percentage points higher than this time four years ago (68%). Not all of those adults will vote, to be sure, but the finding certainly suggests a higher than usual turnout. Shouldn’t categories like first-time voters and self-described non-voters from 2000 be a bit higher than usual rather than lower?

Perhaps Gallup agrees. According to Jones, they will raise their cutoff for likely voters on their last survey coming up this weekend from 55% of adults to 60%. Jones and others with access to the Gallup data tell me such a change would not have altered the results much on the last few Gallup surveys. Perhaps. But if that’s the case, why change now?