Virtually every national survey organization following the presidential race released new surveys in the last four days. Is there any sign of a change during September?
The rolling averages on RealClearPolitics have not changed much, but as they simply average whatever surveys report in a given week, their averages are not always strictly comparable. So I did some of my own calculations this evening.
Seven of the polling organizations that released surveys this week – Gallup, ABC/Washington Post, Time, Fox/Opinion Dynamics, AP/IPSOS, CBS and Zogby – also conducted surveys in the week just after the Republican convention. Using the results available on the Polling Report, I calculated two overall averages of for early and late September, one for “likely voter” samples and another for all registered voters.
In the likely voter samples, Bush led 50% to 43%. His margin is now slightly lower, 49% to 44%, but the differences are not significant even assuming a large pooled sample (note these averages exclude the CBS survey which did not release results for “likely voters”). .
The samples of registered voters looked about the same: Bush led 50% to 43% in early September; he leads 50% to 42% now (Zogby and Fox do not report results for all registered voters).
All but Zogby ask about Bush’s job rating using a scale of approve or disapprove. The average of the six polls in early August (52% approve, 42% disapprove) has not changed significantly (now 52% approve, 43% disapprove).
Thus, once we pool the samples to minimize the effects of sampling error, the vote and presidential job rating have shown little change in September
I will try to explain why in more depth later in the week, but the two numbers I watch most closely are Bush’s job rating and percentage of the vote, as I believe these are most indicative of his ultimate support. That both are hovering just at or above 50 suggest an ultimately close contest, with Bush receiving just about the support he needs to win. The debates will be pivotal.
I, too, track job approval and trial heat polls, but I do it as part of a larger investigation of how much polls lean. You can see a graphical presentation of all the national polls at
http://anonymous.coward.free.fr/polls/pollbias.html.
I agree with you that there is little overall change since the beginning of September, but I’d say that the “path” from the beginning of the month to now wasn’t smooth and steady.
A
fish
Mark Blumenthal: Little Change in September
Mystery Pollster Mark Blumenthal takes a look at the change in poll results over the last month and declares that there has been no meaningful movement. Check it out.
However, he ends his post with this:
I will try to explain why in more depth la…
The debates will be pivotal? CNN just showed the results of a survey they did that shows 80% responded that the debates would have no effect on their vote…
If you go to RealClearPolitics and look at the presidential graph, I think that a significant point can be made by looking at how consistent Bush was during September compared with March through August. During the earlier months Kerry and Bush were moving up and down with no clear lead. It truely was a toss-up during these months. (This may be an artifact of the small number of polls compared to the number in September.) But there is a clear break between the pre-September results and the September results. Kerry should be worried.
If I had to guess I would say that this is a result of Bush not campaigning very hard prior to the Republican convention. I can’t think of anything that has changed so significantly in September as to make a difference other than the campaigning that is taking place now.
Methodology question: why do you feel it is valid to average different polls? Wouldn’t this potenitally magnify the errors inherent in the different methodologies of various pollsters? For example, most polling outfits weights their data according to their own models. All of these models are, presumably, different. Thus, averaging could have a huge and unknown effect on the resulting “average model”. In particular, I would guess that that error would, at the very least, outweigh the benefit (in terms of sampling error) gained by aggregating the polls.
Thoughts?
Friday will be Freedom day. We will be free of the MSM and left leaning bloggers putting all thier tiny eggs into the Debate basket. Best case for Kerry, it a tie. Tie goes to the President.
OT Question for future post —
One of the bits of conventional polling wisdom that I see most often is that polls taken over the weekend show a systematic bias that favors Dems. As a polling layman, this is puzzling. I would think that through some simple regressions, you could quantify this bias and control for it. Could you please comment on this issue as time and interest allow?
In PollingReport.com and other detailed poll reports I haven’t seen something which we report all the time in epidemiology (and which I assume is standard in other branches of social science): that is, response rates.
If N is say, 900, how many potential subjects have to be asked to participate in order to get that number? Has anyone analyzed the properties of the non-responders/refusers?
I, too, have a problem with averaging polls. It is not that it will amplify mistakes, as Doug argues. Averaging the polls could smooth out mistakes just as easily. Here is my problem:
Averaging the polls is effectively a combination of all the different polls into one mega-poll. If you have one poll with 750 people, another with 1,000 people, and a third with 750 people, the average will effectively be a proportionate listing of the opinions of those 2,500 people. The problem with this is that those 2,500 people were asked different questions, asked the same questions in different orders, were asked questions at different times of the day, etc. In other words, these people in this combined uber-poll were in fact subjected to three distinct methodologies. As these different methods can have great affects on the respondents, averaging these 2,500 people together is really averaging apples and oranges.
This is just bad method. I would be skeptical of using it as a “quick and dirty” sense of the race, even…because it commensurates what is effectively incommensurable. If you want to get a sense of the race, you should look at the trend lines in each individual poll. The people in a 9/2 CBS poll are commensurable with the people in a 9/20 CBS poll, as CBS holds its method constant.
If I had to reduce my analysis to just the most critical few numbers, I tend to focus on only two — 1. The % of the vote for Bush; 2. The strong support % for Kerry.
% of the vote for Bush
All along, I have thought that it would be nearly impossible for Bush to do no worse than 45%, yet very difficult for him to get above 48%. The reason is the exceptional support he has received from Republicans and the extremely high opposition from Democrats. For instance, his Republican support is in the 88-90% range. This is incredible when one looks at recent Reuplican candidates. Reagan had 85% Republican support is his 1984 victory (59%-41% over Mondale)and Nixon had 89% support in his landslide over McGovern (61%-39%). (his father’s support was in the high 60’s, low 70’s). Conversely, his opposition is in the low 80’s or his Democratic support is the 18% range, compared with Reagan Democratic support of 34%, Nixon support of about 38% and George H.W. Bush Democratic support in the mid 20’s. In an odd way, George W. Bush has been a uniter. He has united 1/2 of the country in favor of him and 1/2 against him. Even today, after a pretty sucessful last six weeks, he is dropping back to about 50%, and maybe as low as 48%. After this debate, if Bush can get 52% or so, he wins. However, my guess is that the race will narrow even a liitle closer and Bush will be no higher than 48%. Here’s why.
% of strong Democratic support for Kerry
The support for Bush has been very consistent all along and did rise immediately after the RNC (88% – 92%,returning to about 89%. Kerry’s strong support has been a little less. It tended to be in the 78% range and that translated into an overall tie or very slight Kerry lead (he was geting about 60% of the independent and undecided vote). Immediately following the DNC, his lead increased to about 4-5% and his Democratic support increased to about 82-84%.
The Bush campaign then did an interesting thing. They had very little support to gain among Republicans and they weren’t going to increase the Democratic or undecided support for Bush by much, they pursued the addition by subtraction strategy. They attacked Kerry in order to peel away his Democratic and undecided support. It worked and it showed up in the polls. Bush took a 4%-12% lead. More importantly, Kerry’s Democractic support dropped into the 70-72% range and Bush gained about a 60-40 advantage among the undecideds.
Since that time the lead for Bush has shrunk and Kerry’s Democratic support has returned to about the 77-78% range and the undecideds had shrunk to a little more than 50-50.
OK. so what does this brilliant analysis mean? It means that for Bush to go over 50% he’ll need to decrease Kerry’s Democractic support and get about 55% or more of the currently undecided voters. For Kerry to do well and make gains, he needs to convince his softer Democractic supporters that he is a credible and capable Presidential candidate. He doesn’t need to convinced the unconvinced (Republicans) because theyre already voting for Bush. He needs to convince his supporters and give them a reason to really support him (bseides the fact that he isn’t Bush). If he does that, then it is very possible that he can get his Democractic support up to the 82% + range and we’ve got an even race again. Further, history shows that undecided and independent voters tend to select the challenger, if it is a very close race. That’s why they’re undecided. My thinking is that if Kerry re-establishes strong Democractic support he’ll re-establish support from these others.
In summary if at this time next week I see Bush at 52% and Kerry’s strong support at 71%, then Bush probably wins. If I see Bush at 48% or so and Kerry’s strong support back into the low tomid 80’s, then we will have a very interesing October.
PS. A recent Bush strategy is to show an inevitability of his re-election. That strategy is geared toward the Kerry supporters for the purpose of demoralizing them and causing them to essentially give up and withdraw their support for Kerry. Consequently, it is absolutely critical or Kerry to solidfy his support among Democrats in the first debate. If he does that, I think we’ll see polls that are basically tied.
RE: Averaging of polls–not a good idea.
On the other hand, if a range of polls, conducted using a variety of methodologies, point to the same basic situation (in this case Bush up 4 to 6 points), then Bayes Rule suggests you should be more confident in their results.
There is evidence that indicates methodology is driving the difference between the polls. Gallup’s poll shows Bush at 54%, TIPP’s poll shows Bush at 44% (2-way among registered voters). This difference is outside each’s margin of error. Kerry’s number is within the margin of error (44% on TIPP, 41% on Gallup).
Actually, if we look at the numbers Kerry is getting, they range between 46% and 41% in the two-way among registered voters. These differences could simply be the result of normal statistical variation.
But Bush’s numbers vary from 54% to 44%. Statistical variation alone cannot explain these — the maximum variation would be 8%. These numbers must be different because of the different methods the pollsters use.
On the other hand, if you look at the two way likely voter spread, you see Bush’s peak at 52% (Gallup) and valley at 45% (TIPP). That is within the margin of error. Three-way likely voter spread is 52% (Gallup) and 45% (TIPP). This is also within the margin of error. But the difference in the three-way registered voter sample shows a peak for Bush of 54% (Gallup) and a valley for Bush of 44% (TIPP).
So we can say absolutely that taking the mean of the polls with the registered voters is inappropriate. The only way it could be made appropriate is if we start throwing out the outliers (in this case, either Gallup or TIPP). The spread among likely voters is explicable by statistical variation.
So, perhaps I over-stated the problem in my previous post. Nevertheless, there is a methodological problem.
I started writing up a reply to as why an average of polls is superior to one poll. Does anyone know of a reference to this question? Couldn’t we use the Central Limit theorem? In this case the sampling error is inversely related to the sqr root of n where n is the number of polls being averaged. Again however, this requires the same population that is being experimented with. Differing polls technolgies are not the same as conducting identical experiments. However, I am willing to bet a large sum that the co-variance between polls is positive and high and at a minimum we could say that when the Gallup poll goes up, we clearly have a better chance at guessing that the Zobby poll is going up than if we didn’t know the Gallup poll results.
Any takers?
I must firmly agree that an averaging of all poll responses merged together at a point in time, is not all that great a method. Polling days differ, sampling and weighting methodologies differ, and questions differ.
As noted above, I think more instructive is to take individual pollsters and track their trend lines internally. Since they’re (usually) using the same methodologies from iteration to iteration, the conditions for evaluation are relatively stable.
In this vein, we can look at Time’s poll and note that Bush has lost some support compared to the RNC iteration. Gallup is the same way; down from +13 to +8 Bush. IBD has a new one today; Bush goes from up 3 to down 1 among LV. Fox shows essentially a stable number, as do CBS and to a lesser extent WaPo/ABC. Others, such as Zogby, Rasmussen and Harris, see things mostly the way they were at the convention.
Taken together, I think you see a slightly larger “bounce” for Bush than for Kerry, but with a slower and less complete retraction since the convention. This shouldn’t be a surprise; it happens nearly every time in elections, and makes perfect sense: candidate gets largely uninhibited airtime to make his case, support increases. As time goes by and “rebuttals” float in response to that unfettered message, support withers.
I tend to disagree with the notion that Kerry is particularly challenged to turn minds around in the debates. For one thing, the fact that he once was able to command a lead speaks to the idea that those people are likely re-claimable under favorable circumstances. Secondly, it seems clear now that neither Kerry nor Bush definitively sold independents and soft partisans during their conventions. Third, the debates are likely to be more decisive than either convention, given both their greater relative exposure and attention, and their proximity to the actual election.
Thanks for the feedback opportunity, and the solid work in analyzing the situation!
Do pollsters ever ask people about who they voted for in the last election? That would seem to be a good key to this year’s contest — because people who voted for Gore or even Nader in 2000 are likely even more strongly motivated to turn out and vote this year, for Kerry — whereas Bush voters of 2000 may be somewhat less enthusiastic this time around. Does this make any sense?
The trouble with asking who people voted for last time is that a large fraction of voters don’t have any accurate recollection of it.
Normally, they overreport voting for the winner. But not always. Wasn’t there a poll during Watergate in which about two-thirds of a national sample claimed to have voted for McGovern in 1972?
Doug,
An application of the Central Limit Theorem to the aggregate reponses of all the polls would, as you alluded to, require a high level of knowledge about how the poll results are constructed and access to their raw data. Now, some of you out there may have access to such data, but I certainly don’t and the media outlets that pay for the polls genereally don’t release that data. Moreover, even if we disregard the other issues you brought up, to accurately use the Central limit theorem with n=# of polls, n needs to be relatively high and there simply aren’t enough polls being done to create any reasonably accurate estimate.
Now, you may be right concerning the covariance calculation (although we’ve recently had some counterexamples …) but again, I don’t see any way to make a mathematically significant statement concerning the relationship between two polls.
My issue is this: by reporting, for example, a polling average (or some other type of crunched numbers – for an extreme example see Chris Bowers’ “General Election Cattle Call” calculation) gives the appearance of mathematical validity through the simple act of reporting a number. This is a well documented psychological effect (“35% of Americans believe whatever statistics tell them!”) and, in my opininon, does great harm in the public discourse. This is my main beef with the Gallup results – they are being reported as absolute fact without any real discussion of the _potential_ problems inherent in the poll iteself and polling in general.
I still think there is something to be gained by using multiple polls to arrive at a more efficient estimator of the proportion voting for X. I just googled Zellner’s 1961 paper ” An efficient method of estimating seemingly unrelated regressions…” There were at least 384 citations to this seminal work by 1982. This is highly technical, but I would think that the application of this technique to aggregating different polls would yield a more efficient estimator of P, the proportion in favor of Kerry.
For example, if we used regression analysis to estimate that Gallup had a built in positive bias of 10 percent with say an R2 of .95, then clearly we would have a more efficient estimator of both Zoby and Gallup’s estimator when both polls are taken into consideration. When you saw Gallups poll, you would have an estimator of Zoby’s results and vice versa. Using Gallup’s result to solidify Zoby’s estimate would result in reducing the sampling error of each but with the added error of the regression error. Still it would be less than taking either Poll alone. It would still be up to the reader to select either Zoby or Gallup, depending upon their belief in the underlying premise of the polls.
I am just thinking this through as I write this and welcome comments.
I am including my full name as I think there is another Doug posting here.
Notverygeeky,
A Greenberg (D) poll just out showed 50% voting for Bush in 2000 and 41% for Gore. Nader was accurate at 3%. Either people are lying to pollsters (I don’t buy forgetting) or the sample is skewed.
Joe Rozman,
You are absolutely right. Kerry’s strategy from here on out is rally Democrats, not winning over moderate Republicans. And the way to do that is to pound Bush on Iraq. Show that Bush’s judgment on Iraq is poor and Kerry’s is sound. The debate is a perfect chance to do that. If Kerry can state his positions succinctly and blast Bush’s character with respect to the handling of Iraq – no accountability for failures, disconnect from reality, dishonesty, poor judgment – then he will remind Democrats and other Iraq skeptics of why he’s the alternative. I for one date Kerry’s poll slippage to his August 9 Grand Canyon comment about the war resolution. Sure, it may have been accurate but it was stupid. Undecideds hate the Iraq war but they aren’t sure that Kerry does, or that Kerry can right the ship. The debate will go a LONG way toward either convincing undecideds that his stance on Iraq is sound, or that he is hopelessly muddled.
Dogman – the 80% whose minds won’t be changed by the debate don’t matter. The 20% whose minds might be made up, or changed, are where all the action is. According to this morning’s reports on ABC radio, very few of those 20% did actually change their minds.
Elrod – people who hate the war aren’t undecided, they’re voting for Kerry. The people who haven’t made up their minds are more likely to think that getting rid of Saddam was a good idea, but worried that the aftermath will look like Vietnam, and/or who feel betrayed over the WMD issue. I don’t think Kerry’s debate performance last night would change people’s opinions that much, but I’m not an undecided voter.