« August 2005 | Main | October 2005 »

September 30, 2005

Bush Job Approval Up?

Let's continue to obsess over the recent variation on the job approval rating of President George W. Bush and maybe learn something about sampling error in the process.   Two new polls released in the last 24 hours, one by CNN/USAToday/Gallup and another by Fox News/Opinion Dynamics, provide a good opportunity to do both.

Both surveys were conducted this week. The Gallup survey of 1,007 adults was fielded Monday through Wednesday nights (9/26-28; see stories by CNN, USAToday, full results from USAToday and video commentary by Gallup).  The Fox survey of 900 registered voters was done over just Tuesday and Wednesday nights (9/27-28; see story and full results).

  • Gallup shows 45% now approve and 51% disapprove of the president's performance, up from 40% approve-58% disapprove roughly 10 days earlier (9/16-18).
  • Fox shows 45% approve and 47% disapprove, up from 41% approve-51% disapprove 14 days earlier (9/13-14).

Are these trends large enough to be statistically meaningful?  (For the moment, we will look only at the variation due to random sampling, but we need to keep in mind that opinion surveys are subject to all sorts of other sources of error or variability). 

The simplest approach is to look at the published "margin of error" for each survey (3%, assuming a 95% confidence level).   Remembering that the margin of error applies to each percentage separately, it looks as though the "confidence intervals" we get for each result overlap.  For example, the current Gallup approval rating of 45% means we are 95% certain that the result would fall somewhere between 42% and 48% if every American completed the survey.  On the last survey, we are similarly certain the result fell somewhere between 37% and 43%.  Since these ranges overlap, we cannot be certain they are different (the exception is the 7 point drop in Gallup's disapproval rating).

But wait, that's not exactly right.  Each of those reported margins of error tells us how the survey compares to some unknown "true value" for the whole population.  However, when comparing results from two separate surveys, the odds of getting differences of this size by chance alone are a little lower.   Conduct 20 true random samples to measure something and most will cluster near the middle of the normal curve, while extreme values (at the tails of the distribution) are rare. 

So the best way to test for significant differences between two surveys is to use a "Z-test for independence," which requires a spreadsheet or statistical software.  In this case, MP's application of this test shows that the Gallup differences (the comparison between this survey and the last) are significant and the Fox differences are very close.  As MP calculates it, the "p-values" (or probability that the differences occurred by chance alone) are less than 0.03 for the trend in the Gallup approval percentages and less than 0.09 for Fox.  That both surveys show trends in roughly the same direction helps raise our overall confidence.  Looking at it this way, the trend does appear to be statistically "significant."

It also helps that other results provide an explanation for a short term gain.  Americans were most negative about the way Bush handled the response to Katrina in the first few days after the Hurricane hit, but have been more approving since.  On the current Gallup survey, for example, 70% of Americans approve of "the way George W. Bush has handled the response to Hurricane Rita," but only 40% approve of the way he "has handled the response to Hurricane Katrina" (emphasis added).  In a survey conducted September 8-11, Gallup found that only 44% gave Bush a good or very good rating for the way he responded "immediately after Hurricane Katrina hit," but 58% gave him a good or very good rating for his response "in the past few days in the areas affected by Hurricane Katrina and NOT what happened immediately after it hit."

But wait...we are not finished yet.  Take a longer view at the data reported by Gallup and Fox and the trend over the last few weeks and the uptick in Bush's rating appears a bit less significant, both statistically and substantively.  A chart would make this point more clearly, but my software won't let me produce the chart I want, so the table that follows will have to do (for now).   [UPDATE:  Prof. Franklin sent exactly the chart I had in mind -- see below]

0930_job_approval


The main point:  Both Gallup and Fox show slight dips in mid September but higher values on earlier surveys.  For Fox, the mid-September dip was unique - exactly what we might expect given reactions to Katrina.  Gallup, however, shows a similar dip in mid August that is harder to explain.  Are we looking at real change or just random variation during August and September? 

All of this brings us to a third way to consider statistical sampling error.  Suppose for a moment that President Bush's true overall job rating has remained essentially flat over the last month or two (results from other surveys suggest that it has, although it has fallen significantly during 2005).  If that is true, than averaging all surveys should produce a pretty good estimate of the true value.  Since late August, Fox and Gallup both show exactly the same average job rating, 44%, for George W. Bush.  The margin of error tells us that if we take repeated random samples of 1,000 Americans, 19 of 20 should fall within 3% of 44% (or somewhere between 41% and 47%).  Look back at the table for Fox and Gallup, and every survey conducted since late August falls in that range.

So what do we make of all this?  Unfortunately, a certain amount of random statistical noise is an inevitable part of doing opinion surveys.  When we focus too much on small shifts from one survey to the next, we risk mistaking statistical noise for meaningful change.  Given that limitation, the best way to look at polling data is to try to minimize the random variation by considering the results from as many different polls as possible.  Do that graphically (with the charts from Pollkatz or Franklin) and we can be very certain that the President's job rating has fallen during 2005, but less certain about the trend over the last month or two.  Any changes in Bush's overall rating over the six to eight weeks have been small and temporary (although more specific ratings of his leadership have shown bigger drops). 

MP often cringes at headlines that hype small differences between surveys, whether showing Bush going up or going down.  The shifts would often not seem quite so dramatic if compared to previous variation shown by that same survey or if plotted against the results of all other surveys.  The headlines may be as inherent to the process of reporting news as statistical noise is to the survey process, but MP wonders if we can do better.

UPDATE: A picture is worth at least 200 words.  The chart below, kindly provided by Prof. Charles Franklin (aka Political Arithmetik), shows the Gallup surveys as a blue line, the Fox/Opinion Dynamics as a red line and all of the other public polls as grey dots.  Judge for yourself:  Does last upward zig in the blue and red lines represent a significant improvement or just the usual random variation?

Franklin3


Franklin also sent a bonus chart that extends this graphic across the entire Bush presidency.  It makes the same point more vividly.  Take the long view, and the small variations between polls don't amount to much. Note also that both Gallup and Fox tend to fall consistently at the top end of approval band.

Franklin2

Posted by Mark Blumenthal on September 30, 2005 at 05:08 PM in Polls in the News, Sampling Error | Permalink | Comments (2)

September 27, 2005

Gallup's Poll on Polls

Today's video briefing by Gallup Editor-in-Chief Frank Newport had some polling on, of all things, polling.   It raises a few questions that MP finds fascinating.

First, a quick summary of the data from a Gallup poll conducted September 12-15:**

  • 73% of a sample of American adults say the nation would be better off if our leaders "paid more attention to public opinion," and 22% say we would be worse off.
  • 61% on the very next question say the nation would be better off if our leaders paid more attention to "polls," and 33% say we would be worse off.

The "bottom line," according to Newport: "The word 'polls' obviously has a bit more of a pejorative aspect to it, a lower response, than just 'public opinion.'"

Why?  He presents results from two more questions: 

  • "As you may know, most national polls are typically based on a sample of 1,000 adults.  Do you think a sample of this size accurately reflects the views of the nation's population or not?"  30% say yes, 68% say no. 
  • "Some polling organizations make frequent predictions of election results.  What is your general impression of how well they do:  Do you think they are pretty nearly right most of the time, or do you think their record is not very good?"  54% answer mostly right, 41% answer not very good. 

Newport also presents trend data showing that the percentage who said the polls are "pretty nearly right" has fallen steadily since 1985 (from 68%) and was slightly back in 1994 (at 57%) before the advent of most "scientific polling."  Since the 1940s, according to data compiled by the National Council on Public Polls (among others), polling has grown demonstrably more accurate.

 Gallup_poll_on_polls


"Why," Newport asks?

I think part of it may be exit polls.  Remember all the publicity in 2000 and 2004 that exit polls were not accurate?  Well, that may be eroding American's confidence in polling in general, although, I should point out to you all of our analyses of the preelection polls, the kind we do here at Gallup, last year [and] the 2004 elections show they were very, very accurate in predicting the actual popular vote total."

Yes, the obviously widespread perception that the exit polls were wrong in 2004 certainly seems like a reasonable explanation for part of it, except that Gallup's own data show essentially the same confidence in polls now (54%) as in 2001 (52% -- MP knows enough not to claim it "went up" since then).   See the screen shot of the chart above.   The biggest drop registered in Gallup's data occurred between 1996 and 2001.  The unfavorable "publicity" MP remembers most in that period had more to do with election night projections (based on the whole network data collection apparatus) and not just exit polls, but others may remember it differently.

However, MP would nominate another possible culprit:  The perception that polls often show apparently contradictory results.  It is not for nothing that the question about why poll results diverge is #1 on the Mystery Pollster Frequently Asked Question (FAQ) list.  Clearly, some of this perception comes from confusion about sampling error combined with the growing proliferation of public polls.  Some of it also comes from results, even from individual pollsters, that appear to zig and zag in improbable ways.  Gallup is no doubt familiar with such criticism.

While we agree with Newport that the preelection polls of 2004 appear to be as accurate as ever, we should not lose sight of another way to judge the validity of polling data.  How do polls compare to each other?  We have no way, for example, of knowing what the true value is of President Bush's job rating at any given time.  However, when virtually every poll shows a long term, year long decline in that value during 2005, it suggests what methodologists sometime call "face validity."  When one poll fails to show the same trend despite tens of thousands of interviews, it raises important questions about that poll and its methods. 

A bigger issue:  If the public does not trust random sampling (as aptly shown by Gallup's question above) and fails to account for the noise of random sampling error, then all of us who do surveys for a living have a lot more work to do. 

One more thing: 

Remember this survey the next time we discuss whether declining response and coopertion rates are making polls less reliable.   Here's the short version:  The tendency of potential respondents to hang up when called by pollsters only causes greater error (or "non-response bias") if and when those who agree to be interviewed have different opinions than those who do not.   Think about that for a moment.

Now ask yourself:  What are the odds that those who choose to participate in a telephone survey have different opinions on the value of public opinion polling than those who do not? 

Pretty strong, we'd guess. 

**As always, Gallup makes content on its website free to all for the first 24 hours, after that only paid subscribers get access.

Posted by Mark Blumenthal on September 27, 2005 at 04:26 PM in Polls in the News | Permalink | Comments (5)

September 22, 2005

Happy Anniversary

Yes, hard to believe, but it was exactly one year ago today that I wrote my first post on sampling error.  At the time, I was not sure if this experiment would last more than a few weeks, much less a year and counting.  The hundreds of thousands of page visits and views logged by sitemeter are for me, nothing short of amazing. 

I suppose this post would be the ideal forum to reflect a bit on "what it all means," but unfortunately the demands of day job cut into my blogging time.  Tomorrow is the anniversary of when some of you first discovered this site, thanks to those first links from Mickey Kaus and Glenn Reynolds.  So a bit more tomorrow...

Meanwhile, let me offer a huge thank you to all who continue to visit and recommend this site.  It has been a wonderful adventure, and I am looking forward to much more of the same in the coming year. 

Posted by Mark Blumenthal on September 22, 2005 at 05:28 PM | Permalink | Comments (8)

September 21, 2005

Katrina: More from AP-IPSOS & SUSA

Two quick updates, as MP is rushing to catch a train:**

AP-IPSOS has a new survey out this morning conducted September 16-18 (and AP has their usual summary article).  The job rating for President George W. Bush is 40% approve, 57% disapprove, a slight but statistically insignificant improvement from since their last poll two weeks ago (which had his approval drop to 38%).  MP lacks the time to make a graph, but the long term trend in the AP-IPSOS polling is consistent with the other polls.  They show a gradual decline throughout 2005 from 49% in January to 40% today.   The Katrina specific job rating asked by AP-IPSOS, which puts unique emphasis on "the relief effort for victims" is also statistically unchanged at 46% approve, 51% disapprove.

Also, Survey USA's latest automated tracking poll out this morning shows the first big increase in approval of President Bush's "response to Hurricane Katrina" for the first time in weeks, 47% approve, 48% disapprove.   The trend results for this question have shown little movement over the last two weeks, consistently falling with sampling error of 41% approve, 55% disapprove for weeks.   Perhaps this question about "the response to Hurricane Katrina" now also taps attitudes about the response to Rita?

**A reminder that MP will be participating in a forum on polling and blogs hosted by the New York chapter of the American Association for Public Opinion Research tonight in Manhattan.  More details here. The event is open to the general public although an advance RSVP is required and the cost is $20 for non-NYAAPOR members.  Hope to see you there!

Katr

Posted by Mark Blumenthal on September 21, 2005 at 11:59 AM in Polls in the News | Permalink | Comments (3)

September 20, 2005

Katrina: More from Gallup/CNN/USAToday

The Gallup/CNN/USAToday partnership released a new survey last night that shows President George W. Bush’s approval rating at 40%, disapproval at 57%.  Approval of the way Bush is handling “the response to Hurricane Katrina” now stands at a similar 41% approve, 57% disapprove.  Both CNN and USAToday have poll stories, USAToday also provides full results, and as always, Gallup provides a very complete analysis and full results on its web page (newly designed as of this week, but the same subscription policies apply -- David Moore’s analysis will be free to all today, subscription required thereafter). 

As always, the poll is broader in scope than the narrow focus of this blog post may imply.  Read any of the analyses of all that's there, which is considerable.  But for now, let's focus on a few key findings:

1) Bush's job rating.  The USAToday graphic emphasizes the four point drop in approval between the survey Gallup conducted just before Katrina hit (45% on Aug 29-30) to now (40%).  The problem is that Gallup had conducted a poll the week before that showed Bush at roughly the levels seen on the current poll (40% approve, 56% disapprove), and Gallup showed no such decrease in their first two polls after Katrina.  As the chart below shows, Gallup's numbers have been zigging and zagging quite a bit lately, and are not easy to interpret if you assume that every zig and zag is meaningful. 

0920_bush_job_rating


Now, to MP's eyes, this poll to poll variation looks mostly like the sort of random error that is a typical part of the survey process.  We get one big hint of this when we plot party identification (in this case, the percentage who identify or "lean" Republican) against the percentage that approve of Bush's performance.  Note how the lines in the chart below zig and zag in near unison.  While MP cannot know for certain that party identification hasn't been changing in this manner in recent weeks, it seems unlikely.

0920_recent_bush_and_party


The best evidence of this, and the best way to read the Gallup numbers, is to average together their polls to effectively increase the sample size and smooth out the random error.  The chart that follows does this, creating monthly averages for the Gallup poll for all of 2005 (Gallup fields 3 to 4 polls per month).  In this chart, the pattern is smoother and considerably easier to interpret (and also quite consistent with other polls during 2004).  Most important is that the Gallup data, when averaged this way, makes it clear that the big drop in Bush's overall job rating occurred just before Katrina and that his rating has been roughly the same or perhaps slightly worse since (Charles Franklin reached essentially the same conclusion before seeing the latest Gallup numbers).   

0920_yearlong_bush_and_party


Note also the pattern in the average value for the percentage Republican or lean Republican.  Gallup's party identification question (which, unlike many other polls, emphasizes the short term by asking respondents to think about "politics, as of today") shows a slight gradual decrease in Republican identification during 2005.  And, as we would expect, most of the shift appears to be from Republican to independent.  There is an important lesson here in the debate over weighting by party, but MP suspects that those on opposite sides of the debate will reach different conclusions.  We'll come back to that debate another day. 

2) The Katrina Job Rating. The analysis by Gallup's David Moore includes a helpful table that shows how the Katrina specific job rating compares to other issue job ratings of President Bush.  As the table shows (MP recreated it to make it more legible in our format), the Katrina rating is slightly higher than Bush's overall rating, and higher still than ratings of Bush's performance on the economy and Iraq.

0920_specific_job_ratings


All of this leads Gallup's Frank Newport conclude, in his first Gallup Blog post since February (thanks to attentive MP reader "Y" for the tip):

I also suggest caution in accepting the argument that the hurricane caused precipitous damage to the president's standing. Americans had significantly downgraded their assessment of Bush before Katrina . . .

Most available data reinforce the fact that Bush's handling of Katrina is not his greatest weakness at this point. In Gallup's polls, Bush does worse on his handling of Iraq, the economy, foreign affairs, and in particular, gas prices . . .

In short, it is a mistake to assume that the public's mood, views of the economy, views of the top problems facing the country, and views of the administration have undergone profound changes as a result of Hurricane Katrina. The drift toward negativity on these measures was well underway in the late summer. The same problems that faced the nation -- and President Bush -- before the hurricane face him after the hurricane.

True enough.  The Gallup data and the data we have seen elsewhere back him up.  However, there is another way to look at Bush's situation. 

First, if views of the President's handling of Katrina tend to correlate with his current overall job rating, that is not good news for the President.  After all, as the chart above makes clear (and the Pollkatz graphic makes clearer), Bush is currently at a low ebb.  The glass is 60% empty.  Had his job rating been at this level a year ago, the election outcome would have been different.

Second, and far more importantly, if Katrina did not alter Americans overall rating of Bush, they certainly did collapse perceptions of Bush on one key dimension: Being a "strong and decisive leader."   The percentage of Americans who describe Bush as a strong leader fell steadly from 60% just before Katrina, to 51% on the current survey (a result also seen in recent CBS polls).   To paraphrase pollster Peter Hart's conclusion in looking at his own poll NBC and the Wall Street Journal, Katrina effectively "burst" perceptions of Bush as a strong leader.  That may not collapse his overall job rating, but it is a bad sign for the President. 

0920_strong_leader_1


9/22 - Corrected label placement in the strong leader chart above.

Posted by Mark Blumenthal on September 20, 2005 at 10:22 AM in Polls in the News | Permalink | Comments (5)

September 19, 2005

Katrina: New Data and Charts from SUSA

A quick note:  SurveyUSA has continuesd to track reaction to the response to Hurricane Katrina in surveys conducted over the last week.  The latest installment, released this morning, shows approval for President Bush's "response to Hurricane Katrina" at 40%, disapproval at 56%.   

Unbeknownst to MP utnil this weekend, SurveyUSA also recently started posting complete time-series graphics for every question.  Here is how you find the charts (links that follow are for the most recent Katrina poll):  Each SurveyUSA "breaking news" poll report now includes a link at the bottom of the page for "complete interactive crosstabs."  Click that to see demographic crosstabulations for each question.  Clicking on the large white letter "T" on a black background in the upper left corner of each table will generate a time series graph that shows a time series chart for the question, like the one copied below (albeit in tiny form).  Click on the pull-down menu at the upper left of the chart/table and you can modify the base of the chart, changing "all adults" to say, "gender: female" or "race: black."  The "margin of error" at the top of each column changes automatically to take into account smaller subgroup sizes. 

Susa0919



The analysis posted on the SurveyUSA site notes that Bush's approval is "down" two points since his speech on Thursday night, but to MP's eye the recent variation looks mostly like trendless random variation since about September 7.  The 40% approval rating reported today is only one point different than the average of values SurveyUSA has reported since September 7 (41% approve, 55% disapprove).  In fact, MP sees no difference in the average approval rating for the nine days before the speech (9/7-15) compared to the last three days after (9/16-18) - both show 41% approval, 55% disapproval.  Either way, SurveyUSA now provides graphics like this for every question on every one of their breaking news polls, so you can reach your own conclusions.  Just follow the links.

An interesting and more statistically significant trend highlighted in the written analysis appears on the question of whether the federal government is doing too much, not enough or just the right amount to help the victims of Hurricane Katrina.  As this chart shows, the "not enough" percentage has fallen 10 percentage points (from 56% to 46%) since last Tuesday, while the number who say the government is doing "too much" has doubled (from 7% to 15%). 

Say what you will about the sampling issues raised by SurveyUSA's automated interviewing methodology (and MP will have more to say soon), their new graphical charting and cross-tab software is quite unique.  It's worth a few clicks to explore. 

Posted by Mark Blumenthal on September 19, 2005 at 03:01 PM in Polls in the News | Permalink | Comments (4)

A Little Less Mystery

For New York City readers who just can't get enough Mystery Pollster . . .

This Wednesday evening, September 21, MP will participate in a discussion of "Blogging and Polling" sponsored by the New York chapter of the American Association for Public Opinion Research (NYAAPOR).  Michael Cornfield, senior research consultant at the Pew Research Internet & American Life Project, will also be on hand to discuss their data on blogging and the Internet and preview ongoing survey research on bloggers.

The discussion, which runs from 6:00 to 8:00 p.m. (with refreshments at 5:30), will be held at the Fordam University-Lincoln Center.  Please note that attendance is free for NYAAPOR members, $20 for non-members, and an advance RSVP is required).  See the NYAAPOR site for full details (which appear at the bottom of the upcoming events page).

Posted by Mark Blumenthal on September 19, 2005 at 06:25 AM in MP Housekeeping | Permalink | Comments (0)

September 16, 2005

Katrina: Evacuees & Random Samples

Today we have two different perspectives on "random sampling" in gauging reactions to the disaster from Katrina evacuees living in shelters in Texas.  One comes from a true random sample survey conducted by the Washington Post, the Henry J. Kaiser Family Foundation and the Harvard School of Public Health before the last night's televised address by President Bush (article here, full results here).  The second comes from the interviews with Katrina evacuees just outside the Houston Astrodome done live by ABC's Dean Reynolds just after Bush's speech.   

Let's start with the ABC interviews.  The right-wing of the blogosphere is working itself into tizzy over the surprisingly positive reactions from six African-American evacuees interviewed in the parking lot of the Houston Astrodome.   The Media Research Center's Brent Baker provides a concise summary (as well as video excerpts and a full transcript, Ian Schwarz also has a video clip):

Instead of denouncing Bush and blaming him for their plight, they praised Bush and blamed local officials. Reynolds asked Connie London: "Did you harbor any anger toward the President because of the slow federal response?" She rejected the premise: "No, none whatsoever, because I feel like our city and our state government should have been there before the federal government was called in." She pointed out: "They had RTA buses, Greyhound buses, school buses, that was just sitting there going under water when they could have been evacuating people." 

Not one of the six people interviewed on camera had a bad word for Bush -- despite Reynolds' best efforts. . .

Reynolds pressed another woman: "Did you feel that the President was sincere tonight?" She affirmed: "Yes, he was." Reynolds soon wondered who they held culpable for the levee breaks. Unlike the national media, London did not blame supposed Bush-mandated budget cuts: "They've been allocated federal funds to fix the levee system, and it never got done. I fault the mayor of our city personally. I really do."

[Emphasis added].

Now, again, the Post/Kaiser/Harvard researchers fielded their survey of 680 evacuees living in Houston shelters before Bush's speech.  However, the attitudes of their respondents toward Bush and the local officials were quite different: 

  • Only 15% approved of "the way George W. Bush handled the situation caused by Hurricane Katrina;" 70% disapproved.
  • Approval ratings for the governor and mayor were also low, but not as bad as for Bush:  27% approve and 58% disapprove of Governor Blanco's handling of the Hurricane; 33% approve and 53% disapprove of Mayor Nagin's post Hurricane performance. 
  • Evacuees were largely split on who they blamed most.  Roughly the same number blamed the federal government (28%) as blamed either the state of Louisiana (12%) or the city of New Orleans (19%). 

It is worth noting that the survey also shows that ABC interviewee Connie London is not alone in her feeling that, "here in Texas, they have been truly good to us:"

  • Almost all rate the "conditions at their shelter" as excellent (43%) or good (42%).  Only 10% rate them not so good, and only 3% rate them as poor.
  • 82% said they felt "grateful," and 87% that they were "hopeful" about their future.

Now let's consider the methodology of the Washington Post/Kaiser/Harvard study, because at Mystery Pollster, that's what we do.  It was truly a "scientific" random sample survey of evacuees living in Houston area shelters.  As this very complete methodology summary tells us, interviews were conducted in person, from September 10 to12, by professional Houston based interviewers employed by ICR, a survey research company that regularly conducts face-to-face interviews.

Interviewers selected respondents suing a systematic and random procedure that involved choosing every "nth" evacuee.  In other words, they were given a random number -- let's say 20 -- and told to interview every 20th person.  Where evacuees "had limited mobility or were non-mobile" (e.g. sitting on cots or chairs), the interview counted off every "nth" cot or chair and attempte to interview that person where they sat.  Where evacuees were walking around, they stood in a fixed position and attempted to interview every "nth" person that walked by.  The response rate was quite high: 90% of those approached agreed to be interviewed. 

So, MP wonders, how did ABC choose the women they interviewed last night? Obviously, they were not attempting a scientific survey, but did ABC producers select evacuees more or less at random before the speech, asking them to watch and comment afterward?  Or did evacuees gather on their own to watch not knowing Reynolds would interview them later?  Or did they gather near the cameras in hopes of volunteering to be interview subjects? 

MP also wonders, aside from the ABC interviews, just how many evacuees in the Astrodome chose to watch last night's speech?  Moreover, the cynic in MP (and, perhaps, the Democratic partisan) wonders if someone might have tried to coax Bush friendly evacuees to be first in line to be interviewed by ABC's cameras.   

MP hopes an enterprising reporter on scene will try to look into these questions.  It might teach all a lesson or two about the advantages of random sampling . . .  and maybe more . . .

Posted by Mark Blumenthal on September 16, 2005 at 03:52 PM in Polls in the News | Permalink | Comments (4)

September 15, 2005

Katrina: And even more from CBS/NYT, NBC/WSJ, SUSA & Pew

Three new polls out today, plus another report on a poll released a few days ago.  Here are links and a few quick headlines. 

1) A new CBS/New York Times survey is out, the third in three weeks for CBS News.  Results are available in a NYT story and a CBS summary.  Both outlets provide PDF files with full results.  As always, the CBS version tabulates all questions by party identification, the NYT version has extensive time series data and demographics for the current survey. 

On the measure we have been watching closely, 44% now approve of President Bush's handling of the response to Katrina and 50% disapprove, up from 38% approve - 54% disapprove last week.   On the other hand, the surveys shows now change in Americans' unhappiness with the speed of Bush's response: 64% say it was too slow, compared to 65% last week. 

On Bush's overall job rating, the CBS/NYT survey shows 41% approve and 53% disapprove, within a single percentage point of the results they obtained the last three weeks. 

2) NBC News and the Wall Street Journal released a new Hart/McInturff poll, their first since July.  Summary articles are available via MSNBC and the Journal (subscription required), which also has a pdf  online with full results (which seems to be accessible to non-subscribers).

Consistent with virtually every other survey released over the last two weeks, NBC/WSJ finds the overall Bush job rating dropping to the "lowest level of his presidency," in this case 40% approve, 55% disapprove.  The approval rating dropped six points (from 46%) since July.

Their results on Bush's handling on Katrina show on one question, 38% saying they are very or somewhat satisfied with the "way the Bush administration is handling . . . the response to Hurricane Katrina"  58% are very or somewhat dissatisfied).  Then, three question later, 48% approve and 48% disapprove "the way George W. Bush is handling the aftermath of Hurricane Katrina" (emphasis added). 

MP wonders if respondents heard the second question as referring  more specifically to Bush's more recent performance.   Seperately, Gallup found evidence on its most recent survey (subscription required, also available here)  that Americans distinguish the way Bush handled "what happened immediately after Hurricane Katrina hit" from "what has happened in the past few days in the areas affected by Hurricane Katrina and NOT what happened immediately after it hit."  They asked two questions about Bush's handling using that language.  The percentage who rated Bush's performance as "very good" or "good" was 44% immediately after Katrina hit and 58% in the past few days. 

3) SurveyUSA has released several new national surveys in the last few days that update their "breaking news" tracking series.  They released two surveys conducted Tuesday and Wednesday, interviewing 600 adults each evening with their automated methodology that updated their questions on the federal government response to Katrina. 

Unlike the CBS/NYTimes survey, Survey USA has shown no significant trend in the way their respondents rate "President Bush's response to Hurricane Katrina."  On the latest survey, 40% approve and 55% disapprove.  As the chart below shows (apologies to Franklin, his charts are spiffier), the trend has been a flat line over the last week.   

Susa_trend2


[Correction (9/16):  The original version of this post included the wrong chart.  The original chart showed responses to a different question tracked by SurveyUSA which indicates a slight decrease in the percentage (now 51%) that says the  federal government is not doing enough for Hurricane victims].

Separately, SurveyUSA has been conducting another series of national surveys (see the latest) that tracks a slightly different set of ratings of Bush and various federal, state and local officials (using a 10-point numeric scale).  Bush's rating in that series has not changed significantly over the last week either.

4) Finally, the Pew Research Center today released a more complete report on their survey conducted last week.  Pew actually completed two surveys last week, one fielded September 6-7 and a second more in-depth survey completed September 8-11.  They released two reports on the second survey, one on Monday and a longer one today.   Follow the links to find complete questionnaires with results, cross-tabulations and PDF files of the whole package suitable for printing. 

One theme consistent across these surveys is the extent to which Americans see the profound implications of the Katrina for our economy, our image abroad and our ability to respond to a terrorist attack.  A few highlights: 

  • Pew found that 37% of Americans think that "economic conditions in the country as a whole" will be worse a year from now, up from 24% in August and 9% last year. Only 18% say the economy will get better compared to twice as many who held that view a year ago (36%).
  • On a similar question, NBC/WSJ found 49% of Americans saying they expect "the nation's economy" to get worse over the next 12 months, 16% that it will get better.
  • CBS/NYT found that 56% believe "the economy" will get worse "as a result of Hurricane Katrina," only 10% that it will get better.
  • CBS/NYT found that 58% saying that "the federal government's response in the days immediately after the hurricane" has worsened "the United States' image in the rest of the world."
  • NBC/WSJ found that 75% believe we are not "adequately prepared for a nuclear, biological, or chemical attack" (up from 66% two years ago); only 19% believe we are prepared.

As always, MP's brief blog post does little justice to the in-depth analyses from the various pollsters.  If you can, read them in full.

Posted by Mark Blumenthal on September 15, 2005 at 04:41 PM in Polls in the News | Permalink | Comments (0)

September 14, 2005

Katrina Synthesis: Bush Job Rating

On this, MP's tenth post on reactions to the Katrina Hurricane, he feels compelled to remind readers of the continuing human need.  Not that you need reminding.  We know from yesterday's Washington Post/ABC News poll that 60% of Americans say they have already made a contribution for Hurricane relief and another 28% are considering it.  If you happen to fall into the latter category, MP hopes you will consider one of the many worthy charities providing relief.

Now, if there is an upside to this tragedy for those of us who study public opinion, it is the rare opportunity to examine national surveys conducted by eight (or more) pollsters at approximately the same time on the same subject.  We typically consider polling data one survey at a time, and as we all know, any given survey is subject to the vagaries of random sampling.  We often read too much into statistically meaningless results ("Joe Candidate gains a point!").  Even the typical "margin of error" provides only 95% certainty because one survey in twenty will produce a result outside that margin by chance alone.  But if we can look at multiple surveys done at the same time, we can start to see past the random noise of sampling error and come to some very firm conclusions.  The explosion of surveys over the last week provides just such an opportunity. 

As such, I want to take a few posts to review the most clear cut findings consistent across all of last week's surveys on the reaction to Katrina.  Let's start with one of the questions we have been considering, what has the impact been on the job approval rating of President George W. Bush? 

The helpful characteristic of the Bush job approval rating is that most of the pollsters ask it in almost exactly the same way.  "Do you approve or disapprove of the way George W. Bush is handling his job as president?"  Among those that have released job rating results in the last ten days, Gallup, CBS News/NY Times, ABC/Washington Post, the Pew Research Center, Time & Newsweek all ask exactly that question.  AP-IPSOS adds an answer category for those with "mixed feelings" and then probes for whether respondents "lean" toward approval or disapproval.  Zogby asks if they "rate President Bush's performance on the job as excellent, good, fair or poor."   Most of the pollsters also ask the job approval question as the first or second question on the poll, where it is less likely to be influenced by other questions. 

Now obviously the surveys conducted last week were not identical.  While the field dates were very close, they did not exactly coincide.  Minor methodological variation across the various surveys that is largely invisible to consumers (how they train and supervise interviewers, how they select respondents in the household, how the weight the data, and so on) may also introduece some very minor variation into the results.   Also, the Zogby survey screened to interview only "likely voters," the others sampled all adults.  Nonetheless, the surveys conducted last week come as close as we ever to taking multiple "snapshots" of public opinion -- in this case the Bush approval rating -- the same way at the same time.

The best way to consider these surveys is with a chart.  Fortunately, with the help of Poli-Sci Prof Charles Franklin (a.k.a. blogger Political Arithmetik), we have just such a chart (created especially for MP's readers - click the graph for a full size version):

Franklinbush2


As usual, with survey data, the chart shows the evidence of random sampling error and (possibly) some variation resulting from other minor methodological differences.  Yet despite the noise, the lines in the chart tell a clear and consistent story.  Every pollster shows a 3-6 percentage point decline in the Bush Job rating since July to the lowest levels measured during his presidency (for the longer-term view, consider the Pollkatz graphic).  Why does Gallup, the pollster that fielded the most national surveys (8) since July, also appear to show the most variation? MP has no idea, but the more important point is that even Gallup shows a decline since July. 

And the impact of Katrina?  Here, as the chart shows, we have fewer surveys to consider, but the three surveys done just before and after the landfall of the Hurricane suggest that Katrina alone has not had much impact, at least with respect to Bush's overall job rating.  CBS and Gallup showed one point improvements in the approval percentage, ABC/Washington Post showed a three point drop (from 45% to 42%).   None of these taken together were statistically significant and the direction of the differences were obviously inconsistent.

If Katrina alone doesn't explain the most recent drop, then consider the continuing bad news from Iraq combined with the big surge in gas prices over the summer.  Consider also the following ratings of Bush's handling of gas or energy policy.  They are about as negative as job performance ratings get:

  • 72% disapprove "the way Bush is handling the situation with gasoline prices," 25% approve (Washington Post/ABC, 9/8-11)
  • 70% disapprove the way George Bush has been "handling gas prices," 27% approve ( AP/IPSOS, 9/6-8)
  • 73% rate Bush's "handling" of "gas prices" as fair or poor, 21% rate it excellent or good (Zogby, 9/6-7)
  • 76% disapprove of "the way George W. Bush is handling gas prices," 20% approve  (Gallup/CNN/USAToday, 8/28-30)
  • 60% disapprove "the way Bush is handling energy policy," 28% approve (Newsweek, 9/8-9)

Professor Eugene Thiel, (a.k.a. Professor Pollkatz), has been arguing for more than a year that Bush's fortunes closely track the rise and fall of gas prices.  His gas-price graphic (which plots Bush's average job rating against a "reciprocal" gasoline price index in which "up means cheaper") makes a powerful case for his argument (you can find Theil's thoughts on this issue on the main page of his site). 

Itsagas



More to come . . .

-----

Note:  For complete results to the Bush job rating data listed above, see the compliations from the Polling Report or RealClearPolitics (the latter has links) or see my recent posts on these surveys.

Posted by Mark Blumenthal on September 14, 2005 at 05:07 PM in Polls in the News | Permalink | Comments (1)