« November 2005 | Main | January 2006 »

December 25, 2005

Peace on Earth

I will be taking some time off this week to rest, recharge and spend some time with my family.  I'll be back in the New Year.  So from my family to yours, whether you are celebrating Christmas, Hanukka, Kwansa, the New Year or just a few days off to relax, we wish you a season of peace. 

Posted by Mark Blumenthal on December 25, 2005 at 11:03 AM in MP Housekeeping | Permalink | Comments (4)

December 23, 2005


Finally (for today) I have a surprise and something of a holiday gift from the editors of Public Opinion Quarterly.

As a personal venture, this blog has paid some truly gratifying and utterly unexpected dividends.  Topping the list during 2005 was an invitation I received from the editors of Public Opinion Quarterly (POQ) to submit an article for their special issue on the polls, the media and the 2004 campaign.   That issue is out this week, and the editors POQ have consented to allow MP readers to access my article -- "Toward an Open-Source Methodology: What We Can Learn from the Blogosphere" -- free of charge.  There are two versions, a PDF replica of the printed article and an HTML version with live links to footnotes and sources. 

For years I have turned to POQ for the latest gold standard research and commentary on survey methodology from the most noted authorities in the field.  This issue is no exception.  The authors of this special issue include names that should be familiar to regular MP readers including Frank Newport of Gallup, Kathleen Hall Jamieson of the Annenberg School, Kathy Frankovic of CBS and Gary Langer of ABC, as well as others from the worlds of media and academia whose names may not be as familiar but are equally prominent, if not more so. 

The great honor of the inclusion in this special issue is not lost on MP, nor should it be on his readers.  It is a byproduct of the changes that the Internet has produced and of the growing influence of all of you who read and write blogs.  Thank you -- all of you -- for that. 

In their introduction, special issue editors Lawrence Jacobs and Robert Shapiro explain this issue's purpose:

This special issue of Public Opinion Quarterly assembles a wide range of perspectives to critically evaluate polling and American politics in recent election campaigns and the 2004 presidential race in particular. This volume, which follows the 1984 special issue on polls and the media, addresses the major questions that have been raised about election polls today. How accurate are they? Are candidates and government officials poll-hooked addicts who slavishly follow the preferences or whims of the public? Polls and politicians are two elements of a complex process of political strategy and communication. The press is also a critical-or the critical-mediating element. What is the impact of the old and new media's large and ever-growing attention to polling? Does the press serve as a watchdog that ferrets out flawed polls and facilitates strong responsiveness to the public's concern? Or does it do something quite different?

At the urging of the editors, my article provides a review of Internet and blog commentary on polling methodology during the 2004 campaign as well as a review of the emerging methodologies such as automated telephone surveys and Internet polling.  Jacobs and Shapiro also nicely summarize my underlying thesis: 

[Blumenthal] also suggests that the Internet may help make it possible to widen public dissemination of polling results, conduct experiments with surveys, and facilitate a broader and richer scrutiny of polling. New survey methods and modes of public discussion raise questions and challenges for future campaigns.

Again, the editors of POQ have granted permission to MP readers to download my article for free.  While online access to the rest of this special edition is limited to subscribers and AAPOR members, I encourage readers to visit the special issue page and, view the free abstracts and consider ordering a copy of the special issues (a late stocking stuffer perhaps?).  The single issue price is just $10. 

If you would like to order the complete issue, just follow this link.  If you want to order online, go to Single Issues - Personal - Print and click the link for "add to basket."  When the "My Basket" window comes up, select Volume 69, Issue 5.  Ignore the fact that price is listed in British pounds.  I am told that it will display the price in US dollars (or whatever the correct currency is) "at the checkout point" once you enter your payment information.   

If you prefer to order by mail or by fax, you also have the option of starting at the order page, going to Single Issues - Personal - Print and click the link for "$10" which will allow you to download an order form. 

Thanks again to the editors of Public Opinion Quarterly and to all of you for supporting this blog.

UPDATE (12/24): As the article explains, I use the term “open source” as a metaphor to describe the notion of full disclosure of all methodological details, analogous to the way open-source software makes source code available.  Again, as noted in the article, most academic opinion surveys already operate under what is essentially an “open source” model.

However, unlike the idea that Jeff Jarvis floated in April (as he points out in the comments section below), I am not proposing a methodology that would open up survey participation (as a respondent) to all comers.  As commenter Andrew Tyndall also points out in the comments, “opt-in polls are useless as statistical samples.”  Open participation would also raise some profound issues of respondent confidentiality.  However, both Jarvis and Tyndall are asking some provocative questions that are worth taking up in the New Year.

Ah…so many good topics, so little time…

Posted by Mark Blumenthal on December 23, 2005 at 03:44 PM in MP Housekeeping, Polling & the Blogosphere | Permalink | Comments (9)

Iraq Reading List - Update

Not surprisingly, having previously cited a series of academic papers and articles on the relationship between casualties and public support for the Iraq war, I heard from other readers recommending more papers coving similar ground.  I've been so swamped lately that I neglected to pass along some of the links suggested by readers. 

One email came from Richard Eichenberg, a professor of Political Science at Tufts University, who - I just noticed - will be guest blogging at TPMCafe for the next two weeks on public opinion and American foreign policy.

Regarding casualties and support for the Iraq War, Eichenberg recommends:

  • His own paper published earlier this year in the journal International Security entitled, "Victory Has Many Friends," that reviews more than 20 years of public opinion on the use of military force.
  • A similar chapter entitled "The Political Fortunes of War," co-authored by Eichberg and Richard Stoll, along with supplementary material on the impact of casualties on approval ratings.

Michael D. Cobb of North Carolina State University also passes along his paper on "Framing Casualties in the Iraq War," which is currently under review at an academic journal. 

Finally, please note than in my last discussion of this subject, Professors Christopher Gelpi and  Adam Berinsky posted some extra thoughts in the comments section.  Thanks again to both Chris and Adam.

Posted by Mark Blumenthal on December 23, 2005 at 02:51 PM in Iraq | Permalink | Comments (1)

'Twas the Survey Before Christmas...

As things wind down a bit before the holidays, I want to put a little perspective on our recent speculation about the trends in attitudes about President Bush.  It may be helpful to think about two different questions: (1) Have attitudes toward Bush improved since early November?  (2) Have attitudes improved significantly in the week since the election in Iraq?  Surveys have been fairly consistent on the first question (Bush's numbers have improved), but much hazier on the second.  That haziness stems from the usual imprecision of sampling error and timing but also from something few are talking about:  The challenge of conducting a telephone surveys just before Christmas. 

Have attitudes toward the President improved since early November?  Yes.  Every national poll conducted in recent weeks shows at least some increase in the overall Bush job rating since late October.   Although the gain in approval averages to about four percentage points, it ranges from a low of one or two percentage points (NBC/Wall Street Journal and Pew Research Center respectively) to highs of 8 and 11 points (ABC/Washington Post and Diageo/Hotline).  This change and the wide range of results is easier to see in Prof. Franklin's chart, copied below


But have views of the President improved significantly over the last week?  That is a much tougher question.  Only two organizations conducted conventional surveys just before and just after the Iraq election.  Gallup showed a statistically insignificant one point drop in the Bush rating (from 42% to 41%), while the just released Zogby survey showed a six-point gain in the Bush job rating (from 38% to 44%). 

Yet as noted here earlier in the week, two surveys - from ABC/Washington Post and Diageo/Hotline - showed much higher approval percentages (47% and 50% respectively) but neither organization has conducted surveys in early December.  Just this morning, NPR released yet another survey, conducted by Democrat Stan Greenberg and Republican Glen Bolger last Thursday, Sunday and Monday, that has the Bush approval rating at 44%. 

The problem is none of these organizations has polled since at least early November (the last NPR survey was in July), so trying to use these results to judge the trends of the last week requires direct comparisons among results from different firms.  That is always a dicey proposition because of the "house effects" that can cause small but consistent differences across polling organizations.

The Rasmussen automated tracking survey complicates the picture further.  During the month of November, the Rasmussen survey crept up from 43% approval to an average of 45% between December 2-13.    Then during December 14-19 -- the period when both the ABC/Post and Gallup surveys were in the field -- Rasmussen showed approval just a point higher at 46%.  The most recent three day average covering December 20-22 now has Bush up to 50%. 

Professor Franklin has much number crunching and discussion on this subject as well as some new thoughts about the number of days between polls from any one organization can confuse our interpretation of their results.  Over at "The Fix" at WashingtonPost.com, Chris Cillizza has some thoughts as well.   However, MP would argue that given the small number of before and after polls that allow for true apples-to-apples comparisons, it is hard to conclude much except that Bush's overall numbers have probably increased again slightly since last week's elections in Iraq.

But again, we should try to avoid getting so fixated on the day-to-day or week-to-week numbers that we lose sight of the larger picture of public opinion.  First, the recent improvement in the overall Bush ratings are small compared to the year long decline in evidence in the chart above.  Second, on the subject of Iraq, the polls that have tracked attitudes over the course of the year show that attitudes on the whether the war was a mistake or was worth the cost have not varied significantly over the course of the year. 

The data and analysis on Iraq is rich and not easily distilled in a quick blog post, so I would urge interested readers to review the reports from the Pew Research Center and ABC News which go into great depth on attitudes in Iraq.  And if you have time for a third, try the analysis from Gallup (if you have a subscription) or scan their full results (as posted in USAToday). 

Pre-Christmas Effect?

Another important issue to consider is that the challenges facing telephone surveys may be greater at this time of year.  How much greater is a source of some debate among pollsters.  The firms I have worked with have typically advised our clients against fielding polls in the latter half of December (so I have little personal experience with surveys conducted in this period).   Almost every survey organization I know of shuts down altogether between Christmas and New Years.  The reason is that we assume that as the holidays approach, Americans are more likely to be away from home doing shopping, socializing, vacationing or visiting their families.

But in thinking about the recent polls, I realized I cannot remember seeing specific evidence to back this conventional wisdom.  So I went in search of what academic data exists on this subject, and with the help of colleagues on the listserv of the American Association for Public Opinion Research (AAPOR) here is what I can report. 

There are really only two significant studies on this topic.  The first involved research by Gideon Vigderhous published in Public Opinion Quarterly in 1982 (45:2, pp. 250-259).  Vigderhous worked with Bell Canada and he conducted an analysis of year-round customers satisfaction surveys fielded by his company during 1978.  He found that December had the lowest response rates of any month of the year at 40.8%, compared to an average of roughly 57% during the rest of the year.

Twenty years later, five health care researchers who worked on a survey of health issues for the state of Iowa (the Iowa Behavioral Risk Factor Surveillance Survey or BRFSS) set out to gather additional data on this question.  Their study was published in Public Opinion Quarterly in 2002 (Losch, et. al. 66:2, pp. 594-607). They looked at the Iowa BRFSS survey as well as similar health surveys in 29 other states in 1998 and 1999.  They found that the contact and cooperation rates for December were no different than those obtained in other months.  They concluded (p. 606):

These findings cast doubt on the universal recommendation that summer and/or December data collection should be avoided if at all possible.

Unfortunately, this study was limited in one very important way.  All of the December surveys were "completed by the end of the second week of the month" as per the standard practice for most survey researchers.  So the data they examined tells us little about the last ten days or so before Christmas.  "Other studies will need to be conducted," they noted (p. 604), "to fully explore the impact of any 'December' effect."   Sadly, if any such analysis or research exists, MP is not aware of it (at least not as of this posting). 

Armed with what research we do have, let us consider that the two of the most scrutinized surveys -- ABC/Washington Post and Gallup/CNN/USAToday -- were conducted not only at the end of the third week of December but also largely over the weekend.   The Gallup poll was conducted on Friday, Saturday and Sunday.  Calls for the ABC/Post survey were done from Thursday through Sunday.  Did the weekend interviewering make it more difficult to conduct the surveys? 

Unfortunately, the academic are cautionary but not conclusive on this topic.  The most recent looked only at the first two weeks in December.  The Canadian study is more alarming but it is now over 25 years old.  Neither study looked specifically at weekend interviewing in late December. 

I posed the question of polling close to the holidays to the Washington Post's Richard Morin as part of his Tuesday "live chat" on WashingtonPost.com (I was the lucky reader from "Washington DC").  His answer:

I cannot tell you what the cooperation rate for our latest poll because I have not yet received the data. I can't wait to see what it was. My guess is that we found it harder to reach people (lots of no answers) and that cooperation was down a bit. Call it the "Shop-Till-I-Dropped" effect. We did pause before going over [it] this weekend, which is when America--including at least one pollster--did much of its holiday shopping. But the value of going into the field after the Iraq election outweighed these concerns.

Asked to explain the differences between the ABC/Post and Gallup surveys, he added:

When I, just for fun, dropped our Thursday interviews and just looked at the results of our Friday through Sunday subsample, I found that Bush's approval rating was 44 percent. That's still different but far less startling than Gallup's 41. In our poll, Bush had good nights on Thursday and Sunday; his worst nights were Friday and Saturday. A weekend-before-Christmas effect? Perhaps, but no one can say for sure.

I raise this issue not to be critical of Morin or the pollsters at Gallup.  They saw the value of tracking attitudes on an important and developing story and decided that those needs "outweighed" whatever methodological challenges may exist.  As we have seen before, pollsters will sometimes compromise a bit in the face of breaking news.  I also have no particular theory as to whether "weekend-before-Christmas effect" somehow improves or worsens the President's numbers.  However, if the results over the last ten days or so have been a bit more erratic than usual, perhaps some pre-holiday effect is the reason.  Mainly, I wan to issue this caution:  We should assume, until someone produces data to prove otherwise that conditions for telephone polling are more challenging than usual in late December.  Until the next round of surveys appears in early January, we should take the data with a larger than usual grain of salt.   

UPDATE -  Gallup's Jeff Jones emails about their experience with contact and response rates in December:

The last two years we have seen no appreciable change in any of the rates in Dec. compared with earlier months.  We did see some evidence of a slightly lower contact rate in our mid-Dec polls in 2001-2003, however.

Posted by Mark Blumenthal on December 23, 2005 at 12:02 PM in Divergent Polls, Polls in the News, President Bush | Permalink | Comments (6)

December 19, 2005

Not Such An Outlier After All?

The Diageo/Hotline Poll...(story, press release, full results) shows a very different result for the Bush job rating: 50% approve, 47% disapprove.  Is this, as our friend Professor Franklin asks, "a fluke or a harbinger?"  MP guesses it's the former (and Franklin appears to agree), but let's take a closer look.
            -- Me, Yesterday

Finally, [the Hotline's Editor-in-Chief Chuck] Todd points out that for comparison we have only one other conventional poll fielded in the same period.  He has a point.  It is certainly possible that the Fox poll was the outlier on the Bush job rating, or that the reality last week fell somewhere in between.  Since the President made quite a bit of news over the weekend, we may never know for certain.   Nevertheless, if the next round of national surveys shows a surge in the President's job rating back to the high 40s or low 50s, MP may have to concede Todd's point.
            -- Me, This Morning

Advantage, Mr. Todd.  A new national poll is out tonight from ABC News (summary, full results) and the Washington Post (article, full results), and it suggests that the Hotline poll may have been more of harbinger than a fluke after all.  The ABC/Post survey shows the President's job rating at 47% approve and 52% disapprove, up from 39% approve and 60% disapprove on their last survey in early November.

We should keep the field dates in mind, of course.  The ABC/Post survey was conducted from Thursday through Sunday nights last week.  The Diageo/Hotline survey (50% approve) was conducted Sunday and Monday last week.  The Fox News/Opinion Dynamics survey (42% approve) was in the field on Monday and Tuesday nights. 

MP forgot a few things over the weekend.  It is always dicey to compare polls of one organization to those of another, and the "snapshot" for any given week is an imperfect predictor of the snapshot to come next week.  In hindsight, the safest conclusion about the reality of the Bush job rating last week was also the most statistically likely (especially without any obvious skew in partisanship or political ideology in the Hotline and Fox surveys):  Among registered voters, the Bush job approval rating was somewhere in between the levels measured by the Hotline and Fox News polls.  The average of the two was 46% approval, one point lower than what the ABC/Post poll released tonight (47%). 

Approval of President Bush, at least for the moment, appears to have increased significantly in the wake the positive news stories regarding last week's election in Iraq and coverage of five speeches in 19 days.   According to the Post's coverage, the increases came where we might expect in light of the recent communications push:

Bush's pre-Christmas rebound was largely fueled by a sharp increase in support among his core supporters. In the past month, the proportion of Republicans approving of the president's performance increased by nine percentage points to 87 percent. And among conservatives, three in four said Bush was doing a good job, up 12 points from November. Among Democrats, independents and moderates, Bush's support remained unchanged or increased only modestly.

But the Post's Balz and Morin also include this important point :

The other cautionary note for the administration is that Bush's approval ratings and public optimism about Iraq have spiked in the past after instances of positive news, such as the capture of Iraqi President Saddam Hussein or last January's election, only to recede later. However, the gains in the latest poll represent a larger one-time jump than on previous occasions of favorable news from Iraq. Currently, 54 percent say they're optimistic about the situation in Iraq, up from 46 percent a year ago.

Back to my point yesterday about outliers.   To paraphrase from Little Big Man:  Sometimes the magic works, sometimes it does not.  But in the art and science of political polling, it tends to work a lot more often than not.   Franklin's charts make that point vividly.  Yes, outlier's happen, but better to wait to see what the next few polls look (rather than just those from last week) before judging any given poll a "fluke."  Apologies to the Hotline and their pollster Ed Reilly (Financial Dynamics).

LATE UPDATE (12/20):  Hmmm...The polling story this week gets murkier and murkier.  Gallup also released another survey last night (Gallup release, USAToday story & full results, CNN story) and, in apparent contrast to the surveys from ABC/Washington Post and Diageo/Hotline, they show no significant change in the Bush job rating over the last two weeks.  The Gallup poll puts the president's job rating at 41% approve, 56% disapprove, essentially unchanged since two surveys conducted earlier in the month.   

The ABC/Post and Gallup surveys appear to show very different trends in ratings of President Bush's handling of "the situation in Iraq."  The ABC/Post survey shows Bush Iraq approval rating increasing from 36% in early November to 46% now.  The Gallup poll shows his Iraq approval rating at 37% now, just two points higher than on a survey conducted in mid November. 

One possible clue to these differing trends is this line from the ABC News summary:  "Republican self-identification is up six points, to 33 percent of the public."   As of this writing, the Washington Post has not yet released results for party and sample demographics (as it has for other polls in 2005).  On its last two surveys, the ABC Post polls showed a four point Democratic edge in party ID (31% to 27%).   [UPDATE:  The Washington post demographic summary is now online and it shows a one point Democratic edge (33% to 32%) on party identification]. 

Gallup reported party identification for this latest survey at 31% Republican, 32% Democrat and 36% independent.  That mix is roughly the same as their average party identification result for the last three months (32% Republican, 33% Democrat, 34% independent).

Professor Franklin has cruched all of these numbers and he continues to see the Bush job rating from last weeks' Hotline/Diageo poll as a statistical outlier.  Read his post to get all the details, but here is the bottom line.  He uses  statistical modeling to estimate the current Bush job rating at 43.9% and concludes:

[T]he new ABC/WP poll is not inconsistent with the estimate that approval is currently 43.9%. We should expect to see polls this high if approval really is 43.9% and need not conclude that approval has surged to 47%...

I'm still putting my money on the significant upturn to about 42-43%, and holding off on bets in the high 40s. For now.

One more piece of the puzzle to consider:  The Rasmusssen automated survey shows no discernable trend during December.  The current results (46% approve, 52% disapprove) is just one percentage point higher than their average for the month (45% approve, 53% disapprove).  I will be taking a closer look at the Rasmussen trends in posts to follow later this week.

Posted by Mark Blumenthal on December 19, 2005 at 10:35 PM in Polls in the News, President Bush, Sampling Error | Permalink | Comments (2)

December 18, 2005

Outliers Happen

Two new national surveys released late last week add yet another wrinkle to the question of whether the approval ratings of President Bush are on the rise since late October, because they are at once very similar in methodology and yet yield a extremely different result for the Bush job rating.   The latest Fox News/Opinion Dynamics poll (story, full results) reports results roughly comparable to other recent surveys.  They put the Bush job rating at 42% approve 51% disapprove (essentially no change over the last two weeks).  The Diageo/Hotline Poll, however (story, press release, full results) shows a very different result for the Bush job rating: 50% approve, 47% disapprove.  Is this, as our friend Professor Franklin asks, "a fluke or a harbinger?"  MP guesses it's the former (and Franklin appears to agree), but let's take a closer look.


Let's consider some factors that might explain the difference:

Registered voters?  Most national polls report the Bush job rating among all adults.  The Diageo/Hotline poll surveys self-identified registered voters.  However, so did the Fox poll.  Moreover, in other surveys, the results for the Bush job rating are not much different among registered voters (not surprising, as one universe is typically at least 85% of the other).  For example, on the ten surveys fielded by the Pew Research Center during 2005, the Bush rating was only two percentage points higher among registered voters (44%) than among all adults (42%).  According to Gallup, their most recent poll showed Bush approval one point higher among registered voters (43% vs. 43%), and over the course of the year, "you might see a point difference here or there, but for the most part the adult number is the RV number" (thanks to PRC & Gallup for the data). 

Party Identification?  A big difference in party identification between the November and December Diageo/Hotline polls might suggest some random variation in the composition of the sampled respondents that would explain the jump in the Bush rating.  However, the differences in party identification are small and non-significant.  If anything the most recent Diageo/Hotline poll is less Republican (30% Republican, 28% Democrat) than the last (33% GOP, 28% Dem).  With independent "leaners" included, the GOP or lean GOP portion fell from 38% to 35% between the two polls, while the Democrat/lean Democrat portion held steady at 37%.  The party balance with leaners included is similar to the Fox Poll, which shows 37% GOP-39% Dem (on a question that does not offer an independent category as a possible answer). 

As Charles Franklin points out, the Bush approval rating in the Diageo/Hotline poll is up across all three party groupings, Democrat, Republican and independent:

This poll finds approval of 86%, 44% and 17% for Reps, Inds and Dems. Those are quite a bit higher than other polls have found recently, but are up across the board in this sample. Gallup's most recent results were 81%, 38% and 10% approval by Reps, Inds, and Dems.

So the difference in this poll is not about the mix of Demcrats and Republicans but about the level of Bush support measured across party lines. 

Other "diagnostics?  Nothing obvious stands out to distinguish the November and December Diageo/Hotline samples.  Both have essentially the same composition in terms of gender, age, education and race.  The December sample is slightly more conservative (37% vs. 34%), "born-again or evangelical Christian" (41% vs. 37%) and has Bush beating Kerry by a wider margin (49% to 37%) than the last survey (46% to 41%) in self-reported recall of the 2004 vote (remember, survey self-reports always overstate the victory margin for the winner).   None of these small differences are large enough to attain statistical significance.  While the slightly more conservative cast probably contributes to the difference in the Bush rating, it cannot explain all of it. 

Question order?  Most of the other national surveys, including Fox/Opinion Dynamics, ask the presidential job rating as either the first or second question on the survey.  Unlike the other surveys, the Diageo/Hotline polls asks the presidential job approval rating after a battery of favorable ratings of public figures.  Of course, it had as similar battery on previous polls, and those showed the Bush job rating at about the same level as other surveys.  Looking at the questions asked before the job rating, two things changed from November to December.  They cut favorable ratings for Dick Cheney, John Kerry and the U.S. Congress and replaced them with ratings of Randy "Duke" Cunningham, Jack Abramoff and John Murtha.  I agree with Franklin that there is little chance the "priming" effect of these questions explains the better result for Bush on this survey.

The Diageo/Hotline poll also moved the party identification question from the beginning of the survey to the end.  There is some experimental evidence that asking the party ID question first resulted in a lower job approval rating during 2004.  However, the difference was only about 2% points, and of course, nearly every other pollster asks party ID toward the end of the questionnaire.   So while that change may have contributed a point or two, it cannot explain why the Bush approval rating on this survey is so much higher than the other polls. 

Survey dates?   The Diageo/Hotline poll was conducted Sunday and Monday last week, while most of the other surveys were fielded the previous week.  Could their result indicate a late Bush surge?  The problem with that theory is that the Fox survey was fielded Monday and Tuesday nights last week.  The Rasmussen automated survey - which typically shows the Bush job rating to be a few points higher than other surveys - had it at 44% on Sunday to Tuesday nights last week, which is about a point lower than their average over the last two weeks.  So if the Hotline poll is catching a late surge, it is not evident in two other polls conducted at the same time.

Sampling Error?  The reported "margins of error" are 3.4% for the Diageo/Hotline poll and 3% for the Fox News poll.  That means that in theory, the true underlying population value should fall within that range for each survey.  That would mean that the "true" Bush approval rating should fall (with rounding) between 47% and 53% and between 39% and 45% for the Fox poll.  Since those ranges do not overlap, the two polls show a "statistically significant" difference.

But keep in mind that these "margins of error" are based on a confidence level of 95%.  That means that we might expect the "true" number to fall outside that range on at least one poll in twenty (or one question in twenty within any survey) by random chance alone.  Pollsters use the word "statistically significant" to describe any difference that we are at least 95% confident about.  But such differences can still occur, in rare cases, by random chance.   It is entirely possible that the Diageo/Hotline approval rating in this case is just such a rare statistical fluke. 

While Typepad was down today, I emailed Charles Franklin and asked him to check his database of Bush approval polls to see if he could find similar "outliers" from the last few years.  He did so and has some new graphics showing that, yes, "outliers happen."  Check the following graph.  The Diageo/Hotline poll is the last red dot on the right side of the chart.


Franklin found that the Diageo/Hotline poll is among the "most extreme 1%" of all polls in terms of its deviation from what other polls are showing at roughly the same time.  But what do we make of that?  Here is Franklin's take:

But every dataset has to have a most-extreme 1% of cases, so what can we make of this result? One conclusion is that we should thank the Diageo/Hotline people for reporting their results despite the large difference from other polls. It is not unheard of for pollsters to bury their results that look too different, or to "fiddle" with the weighting or other things to bring the results a bit closer to other polling. Everyone wants to make news but no one wants to look too extreme. It is, ironically, a sign of the credibility of the Diageo/Hotline people that they were willing to put this result out when it was certain to be remarked upon as out of line with other polling (exactly as I am doing here!).

But the fact remains that this poll is far beyond the bounds we would normally expect, even when taking total survey error into account, and not just sampling error. By this standard, we can say that we would very much doubt that the Diageo/Hotline result is simply a random outcome from the same process that has generated all the other polling. But the source of that exceptional variation remains a mystery. Our conclusion should be that this result should be substantially discounted in estimating approval of President Bush.

Every polling organization can produce results that are outliers. What is important is spotting them and putting them in proper perspective. That is far more desirable than suppressing the results or pointing to them as examples of "bias". What matters is performance over the long term, not in any single sample.

Here, here.  Hear, hear[Thanks Michael B]

UPDATE (12/19):  Chuck Todd, The Hotline's Editor-in-Chief, emailed to chide MP for focusing on only one question from their poll.  That is a fair criticism, so let us note that the Diageo/Hotline survey also showed:

  • A "crash" in the job approval ratings of the U.S. Congress. Approval fell ten-points (from 36% to 26%) and disapproval rose 14-points (from 50% to 64%) since November.
  • A ten-point Democratic advantage (43% to 33%) on the "generic" Congressional vote preference, representing a net Democratic gain of four points since November (when voters preferred Democrats 41% to 35%).
  • A six-point increase in the favorable rating of the Democratic Party (from 44% to 50%) and a six-point increase in the favorable rating of the Republican Party (from 41% to 47%). 

Also to be clear, I am not arguing above that the Hotline poll was skewed in the Republicans' favor.   An outlier result for party identification would certainly make for better support for Republicans throughout the survey, and obviously, that was not the case.  However, the point I was trying to make above - and may not have made very well - is that it is possible, in very rare instances, to get an extreme "outlier" result for just one question on a poll even when results to other questions appear representative. 

Finally, Todd points out that for comparison we have only one other conventional poll fielded in the same period.  He has a point.  It is certainly possible that the Fox poll was the outlier on the Bush job rating, or that the reality last week fell somewhere in between.  Since the President made quite a bit of news over the weekend, we may never know for certain.   Nevertheless, if the next round of national surveys shows a surge in the President's job rating back to the high 40s or low 50s, MP may have to concede Todd's point.

Posted by Mark Blumenthal on December 18, 2005 at 01:57 PM in Polls in the News, President Bush, Sampling Error | Permalink | Comments (6)

December 16, 2005

Typepad Outage

A quick update for those wondering why all of my posts from this week seemed to disappear overnight:  Typepad, the service that hosts this humble blog, experienced a major service outage overnight that prevented me from posting and readers from commenting.  During the day Typepad also "deployed" a backup copy of all of its weblogs from earlier in the week, which is why this week's posts seemed to disappear. 

As of now, all of my posts have been restored, but some of the images posted this week are still missing.    Typepad promises to restore the images over the weekend.  Hopefully all will be back to normal by Monday. 

For those interested, see the initial explanation and status updates from Typepad and coverage from MSNBC, ZDNet, Information Week and Computer World.  Unfortunately, such growing pains are inevitable as the blogosphere continues to expand.   Frustrating as an outage like this has been, my experience with Typepad has been mostly positive.  As Glenn  Reynolds points out, "it's to their credit that an outage like this is rare enough to be big news."

Posted by Mark Blumenthal on December 16, 2005 at 08:45 PM in MP Housekeeping | Permalink | Comments (1)

December 15, 2005

Bush Bump-Up Stall?: Pew, NBC/WSJ & Zogby

Yesterday saw releases of two new national surveys by the Pew Research Center (report, questionnaire, tables) and NBC/Wall Street Journal (full results & coverage by WSJ and MSNBC).  Combined with a new Zogby survey released on Monday, we now have eight conventional telephone surveys conducted in December.  Although seven of eight show a higher approval rating for President Bush, the three most recent releases all show Bush's job rating under 40% and indicate smaller increases since November than the other four.  As the field dates largely overlapped, the timing of the polls does not appear to explain difference.


As usual the best way to consider the trend is to see these results plotted graphically, and once again Prof. Charles Franklin's updated graphic provides the most coherent view:


The regression lines that Franklin plots through the data take into account the "house effects" that make Bush's job rating typically run a little higher on some polls and a little lower on others.  He explains those lines better than I can, and his analysis is worth reading in full.  However, here is Franklin's bottom line: 

[T]here is reason to wonder if the significant rise we've seen since Veterans day has now reached a plateau, at least for the moment...approval continues to be significantly higher than the low-points before Veterans Day, but there is at least a reasonable doubt that it is continuing up at the rate it has for the past 3-4 weeks.

Also as usual, the Pew Research Center analysis goes into great depth on Iraq and a variety of other issues, including graphical views of the data that are well worth the click.  On Iraq, the Pew Center sees more continuity than change:

[F]undamental public attitudes toward the war have not been changed  in either direction  by the clashing points of view. Pew's latest national survey shows that the public continues to be evenly divided about whether to withdraw U.S. forces as soon as possible or keep them in Iraq until the country is stabilized, as well as over the decision to take military action in Iraq.

The most interesting finding comes from the way Pew combined results from questions that look into prospective policy regarding Iraq.  As noted here previously, questions on what the U.S. should do next in Iraq yield highly inconsistent results across surveys.  The new Pew analysis shows why.  They found the nation remains divided on a question tracked since the beginning of the war: Whether the US should "should keep military troops in Iraq until the situation has stabilized" (49%) or "bring its troops home as soon as possible" (46%).  However, they added a follow-up question on the recent survey, asking those who want to bring the troops home whether they favor an immediate or gradual return.  They also looked at whether those who want to keep troops also want to "set a timetable for when troops will be withdrawn."  Their finding: 

The roughly even division in the public over whether to keep troops in Iraq obscures a more complicated set of opinions about what to do next. Most of those who want to bring troops home "as soon as possible" apparently do not mean "now," and not everyone who wants the U.S. to stay in Iraq is opposed to setting a timetable for withdrawal.

Of those who support bringing the troops home, most favor a gradual withdrawal over the next one to two years rather than an abrupt departure. Even among liberal Democrats, 66% of whom favor disengagement, most believe this withdrawal should be gradual (40% favor gradual withdrawal, 24% think it should occur immediately). Within every partisan group across the spectrum, support for bringing troops home is more likely to mean gradual rather than immediate withdrawal.

See the report for complete data and question wording. 

Posted by Mark Blumenthal on December 15, 2005 at 06:19 AM in Polls in the News, President Bush | Permalink | Comments (4)

December 14, 2005

Zogby, Wal-Mart & Ethics

Last week, syndicated columnist Joel Mowbray took pollster John Zogby to task in the Washington Times for "serious ethical issues which call into question the integrity of the much-ballyhooed survey" on the retailer Wal-Mart.  Zogby's poll, which had been commissioned by a labor-backed group known as WakeUpWalMart.com, reported among other things that 56 percent of Americans agreed with the statement, "Wal-Mart is bad for America."   While MP has never been a big fan of Zogby's methodology, in this case at least, the criticism seems a bit unfair. 

Some background:  MP will leave for another day an in-depth look at Mr. Zogby's colorful history and unique approach to survey methods.  For now, suffice it to say that MP has long been skeptical of Zogby's practices of interviewing during daytime hours and sampling from listed telephone directories, adjusting samples by party to match past exit polls or "his judgment of 'what is happening on the ground.'" The amazing last minute "corrections" that brought his results for the 2004 New Hampshire and Georgia primaries into line with those of other pollsters and the ill-fated Kerry victory projection made at 5:00 p.m. EST on Election Day (a projection which neatly coincided the results of exit polls that had been widely leaked earlier that afternoon) did not exactly bolster my confidence (more details here, here and here; use CTRL-F to find "Zogby"). 

Now before delving into the specifics of Joel Mowbray's argument, it is worth noting that Zogby has also been criticized in the past for failing to disclose the identity of clients who paid for results released into the public domain.  For example, consider this passage from last year's classic Zogby profile by the New Yorker's Larissa MacFarquhar:

Zogby is also willing to guard the identity of his clients if they want to remain anonymous, while AAPOR [the American Association for Public Opinion Research] insists that an ethical pollster will always make the sponsor known if a poll's results are published, so that the public can judge whether or not he had a vested interest in the outcome. Zogby argues, reasonably enough, that if his methods are transparent and correct, the sponsor should be irrelevant. He tells clients that if they wish to publish any part of a survey they have to publish the whole thing, including the questions. But if a client wishes to publish only part of a survey-the results that he likes-Zogby offers another, rather underhanded option: he will conduct a second survey, using only the questions from the first one whose answers served the client's purposes, and the client can publish that and keep the first poll private.

In the case of the Wal-Mart poll, however, disclosure of the client's identity was not an issue. The various press releases and the articles by the Associated Press and Reuters all clearly stated that the poll was "commissioned" or "released" by WakeUpWalMart.com.

Yes, Zogby was hired by the interest group in question, but he is certainly not the first pollster to work on behalf of a private interest.  Results from polls conducted for political campaigns and interest groups are released almost every day.  The pollsters that conduct that research -- including  MP -- operate under essentially the same ethical guidelines as all survey researchers.  At the risk of oversimplifying AAPOR's Code of Professional Ethics and Practices, those obligations include taking "all reasonable steps" to measure public opinion fairly and accurately and then, if releasing the results publicly, describing the results honestly and disclosing both the sponsor and "essential" details about methodology.  Full disclosure of sponsorship allows the public to take privately funded research with the appropriate grain of salt. 

So if in this case Zogby fully disclosed the sponsor, what was Mowbray's beef?  He alleges two specific failings:

1)  Before conducting this poll, Zogby had done other work for anti-Wal-Mart interests:

In recent years, Mr. Zogby has pocketed roughly $90,000 to serve as an expert witness for individuals suing Wal-Mart, according to testimony he gave in a deposition last year in an Arizona case. Nowhere is Mr. Zogby's prior work on behalf of plaintiffs mentioned in the press release announcing the poll results.

Had Zogby hidden WakeUpWalMart's sponsorship of his poll, Mobrey would have had a point, but I am not sure what the reader gains from knowing that Zogby also did other work for essentially the same client.  Mowbray claims that "what no journalist would have known without digging is that Mr. Zogby cannot be considered an objective third-party when it comes to Wal-Mart."  If payments from an anti-Wal-Mart interest compromise a pollster's "objectivity," did journalists really need to dig farther than the press releases disclosing who sponsored the poll?   A researcher's prior work may be relevant, but asking every pollster to disclose all work done for the same interest that is paying for the poll seems over the top.  It is certainly not required by the AAPOR Code. 

2)  Zogby's client wrote the first draft of the press release on the results

Though Mr. Zogby insisted that being paid tens of thousands by people suing the retailer did not compromise his objectivity, he was careful to note that the press release announcing the poll results was drafted by the client, Big Labor-backed WakeUpWalMart.com. But when reached the following morning, Mr. Zogby conceded that his staff "heavily edited" the release and even posted it on the group's Web site and put the release out over its wire.

MP will grant that letting a client draft the press release that goes out over the pollster's letterhead is not something that most in our field would consider a "best practice." However, from Mowbray's account, it is not clear that is what happened here.  The press release posted on the WakeUpWalMart site is different than the one that appears on Zogby.com.  Which release did Zogby attempt to distance himself from?  Which did Zogby's staff "heavily edit?"  I know of no pollster code of ethics - including the AAPOR code - that would prohibit a client from putting out a press release put out on their own letterhead.

There is an important ethical question raised here, but one not quite as dramatic nor as theoretically damaging to this particular poll.   It involves Zogby's habit of conducting polls for both mainstream media outlets and political interest groups.    Mowbray is probably right to assume that WakeUpWalMart hired Zogby because of his "extra panache and an air of instant credibility of his reputation."  That reputation comes as result of his high profile media polls.  Consumers of Zogby's media-sponsored surveys have the right to ask whether his work for private interests clouds his objectivity.  Needless to say, Zogby argues vehemently in his response to Mobray that it does not. 

This is a conversation worth having, but we should be fair to John Zogby.  He is certainly not the only "public" pollster whose client list also includes interest groups or partisans.  His approach may be a bit more flamboyant and unorthodox -- to put it mildly -- but on this general ethical issue he is not alone. 

Interest disclosed:  I conduct "internal" polls for Democratic candidates and interest groups, some of which get released into the public domain.  I have never conducted a survey for a news organization. 

LATE UPDATE (12/15):  The Pew Research Center just released a survey (report, full results, tables) that probed attitudes on Wal-Mart, including favorable ratings for Wal-Mart and Target that appear to be based on a very similarly worded question.  In the Pew survey, 65% of Americans have a favorable opinion of Wal-Mart, 30% have an unfavorable opinion.   On the Zogby survey, 58% of Americans rated Wal-Mart favorably and 38% unfavorably. 

The Pew and Zogby surveys were more in synch in measuring opinions of Target:  Pew reported a 76% favorable, 14% unfavorable rating.  Zogby reported 73% favorable, 13% unfavorable rating.

The Pew survey had much more to say about perceptions of Wal-Mart:

Nearly every American lives near enough a Wal-Mart to shop there, and 84% say they have done so in the past year. Praise for the retailer's low prices, wide selection and convenience flow freely, and 81% of those with a Wal-Mart nearby say it is a good place to shop.

Somewhat less glowing, however, are judgments about Wal-Mart's effect on communities and the nation as a whole, and a third of the public (34%) rates it a bad place to work. Overall, 69% of those familiar with Wal-Mart have a favorable opinion of the company. Still, 31% have an unfavorable view, which is a considerably higher negative rating than is accorded to many other major corporations

Posted by Mark Blumenthal on December 14, 2005 at 06:15 PM in Polls in the News | Permalink | Comments (2)

December 12, 2005

Bush "Bump Up" Update: Gallup/CNN/USAToday

The National Journal is reporting this morning, via the (subscription only)  Hotline Wake-Up Call, that the latest survey from Gallup shows:  " Pres. Bush's approval rating at 43% among adults, up from 38% in late 11/05. 52% disapprove."   Yesterday's video briefing by Gallup's Frank Newport again promised the release today of results from than 2,000 interivews. 

Thus, we have three conventional polls (Fox, CBS/NYT and Gallup) that confirm the trend in the Rasmussen automated survey. 

Update (12/13):  USAToday now has a story and full results posted, as does CNN.  Interestingly, Gallup released Bush job rating results from two different surveys on Monday.   The early report, showing Bush with a job rating of 43% approve, 52% disapprove came from a survey of 1,013 adults conducted last week (12/5-8).   The results released later by CNN and USAToday -- showing Bush with a job rating of 42% approve and 55% disapprove -- came from a second survey of 1,003 adults conducted from Thursday to Sunday last week (12/9-11).

Gallup has not yet posted their official summary of the second survey.  However, readers should note that first survey was a bit more Republican than others Gallup has reported recently.  The composition of the first survey was 36% Republican, 31% Democrat, 31% independent.   The percentage of GOP identifiers was the most on a Gallup survey since February.   Other Gallup surveys released since  September have  reported the GOP percentages between 30% and 33%.

The story by USAToday's Susan Page offered this analysis of what may be driving the increase in the Bush ratings:

Bush's rating rose by 3 to 6 percentage points for his handling of a half-dozen specific issues, though he gets majority approval only on the issue of terrorism. Most Americans disapprove of his handling of the economy, by 58%-40%, and of the situation in Iraq, by 59%-39%.

His biggest jump was a 6-point rise in approval for his handling of gas and home heating prices — still a dismal 28% approve-67% disapprove.

The CNN story notes that the Iraq rating was up three points over the same period:

Nearly three out of five Americans, 59 percent, said they disapproved of the way Bush is handling Iraq; 39 percent said they approve.

The approval number is up slightly from last month, when a CNN/USA Today/Gallup poll found 63 percent of Americans disapproved of the administration's Iraq policy and 35 percent said they approved. In September, 32 percent said they approved of Bush's handling of Iraq.

Late Update:  Gallup's official summary of the 12/9-11 survey by David Moore is now up along with a video briefing by Frank Newport (both should remain free to non-subscribers until Wednesday) .   Both offer a "broader interpretation" of what is driving the recent increase in the overall Bush job rating.  Here is Moore's take: 

On the economy, Bush's rating is up three percentage points since the Nov. 11-13 poll (37% to 40%); on Iraq up four points (35% to 39%); on terrorism up four points (48% to 52%); and on foreign affairs up five points (37% to 42%).

Other polls have also shown an increase in Bush's standing among the public, with some commentators suggesting the improvement is related not to Iraq but to the economy. The poll results here suggest perhaps an even broader interpretation -- that Bush's improvement may be related to his making a more aggressive defense of his administration, regardless of the specific issue. With slight to modest increases across the various issues noted above, it would appear that many Americans (mostly independents) have more positive views of the president overall, not just on the way he is handling the war in Iraq.

Also, MP overlooked a new AP-IPSOS survey released last Friday (conducted 12/5-7 among 1,002 adults, topline results here and here) that showed the Bush job rating increasing to 42% from 37% in early November.

Finally, the Cook Political Report and RT Strategies released results from a new survey (topline and press release) conducted among 1,000 adults from December 8-11 showing the Bush job rating at 42% approve, 55% disapprove. 

Posted by Mark Blumenthal on December 12, 2005 at 10:51 AM in Polls in the News, President Bush | Permalink | Comments (4)