« Katrina: And even more from CBS/NYT, NBC/WSJ, SUSA & Pew | Main | A Little Less Mystery »

September 16, 2005

Katrina: Evacuees & Random Samples

Today we have two different perspectives on "random sampling" in gauging reactions to the disaster from Katrina evacuees living in shelters in Texas.  One comes from a true random sample survey conducted by the Washington Post, the Henry J. Kaiser Family Foundation and the Harvard School of Public Health before the last night's televised address by President Bush (article here, full results here).  The second comes from the interviews with Katrina evacuees just outside the Houston Astrodome done live by ABC's Dean Reynolds just after Bush's speech.   

Let's start with the ABC interviews.  The right-wing of the blogosphere is working itself into tizzy over the surprisingly positive reactions from six African-American evacuees interviewed in the parking lot of the Houston Astrodome.   The Media Research Center's Brent Baker provides a concise summary (as well as video excerpts and a full transcript, Ian Schwarz also has a video clip):

Instead of denouncing Bush and blaming him for their plight, they praised Bush and blamed local officials. Reynolds asked Connie London: "Did you harbor any anger toward the President because of the slow federal response?" She rejected the premise: "No, none whatsoever, because I feel like our city and our state government should have been there before the federal government was called in." She pointed out: "They had RTA buses, Greyhound buses, school buses, that was just sitting there going under water when they could have been evacuating people." 

Not one of the six people interviewed on camera had a bad word for Bush -- despite Reynolds' best efforts. . .

Reynolds pressed another woman: "Did you feel that the President was sincere tonight?" She affirmed: "Yes, he was." Reynolds soon wondered who they held culpable for the levee breaks. Unlike the national media, London did not blame supposed Bush-mandated budget cuts: "They've been allocated federal funds to fix the levee system, and it never got done. I fault the mayor of our city personally. I really do."

[Emphasis added].

Now, again, the Post/Kaiser/Harvard researchers fielded their survey of 680 evacuees living in Houston shelters before Bush's speech.  However, the attitudes of their respondents toward Bush and the local officials were quite different: 

  • Only 15% approved of "the way George W. Bush handled the situation caused by Hurricane Katrina;" 70% disapproved.
  • Approval ratings for the governor and mayor were also low, but not as bad as for Bush:  27% approve and 58% disapprove of Governor Blanco's handling of the Hurricane; 33% approve and 53% disapprove of Mayor Nagin's post Hurricane performance. 
  • Evacuees were largely split on who they blamed most.  Roughly the same number blamed the federal government (28%) as blamed either the state of Louisiana (12%) or the city of New Orleans (19%). 

It is worth noting that the survey also shows that ABC interviewee Connie London is not alone in her feeling that, "here in Texas, they have been truly good to us:"

  • Almost all rate the "conditions at their shelter" as excellent (43%) or good (42%).  Only 10% rate them not so good, and only 3% rate them as poor.
  • 82% said they felt "grateful," and 87% that they were "hopeful" about their future.

Now let's consider the methodology of the Washington Post/Kaiser/Harvard study, because at Mystery Pollster, that's what we do.  It was truly a "scientific" random sample survey of evacuees living in Houston area shelters.  As this very complete methodology summary tells us, interviews were conducted in person, from September 10 to12, by professional Houston based interviewers employed by ICR, a survey research company that regularly conducts face-to-face interviews.

Interviewers selected respondents suing a systematic and random procedure that involved choosing every "nth" evacuee.  In other words, they were given a random number -- let's say 20 -- and told to interview every 20th person.  Where evacuees "had limited mobility or were non-mobile" (e.g. sitting on cots or chairs), the interview counted off every "nth" cot or chair and attempte to interview that person where they sat.  Where evacuees were walking around, they stood in a fixed position and attempted to interview every "nth" person that walked by.  The response rate was quite high: 90% of those approached agreed to be interviewed. 

So, MP wonders, how did ABC choose the women they interviewed last night? Obviously, they were not attempting a scientific survey, but did ABC producers select evacuees more or less at random before the speech, asking them to watch and comment afterward?  Or did evacuees gather on their own to watch not knowing Reynolds would interview them later?  Or did they gather near the cameras in hopes of volunteering to be interview subjects? 

MP also wonders, aside from the ABC interviews, just how many evacuees in the Astrodome chose to watch last night's speech?  Moreover, the cynic in MP (and, perhaps, the Democratic partisan) wonders if someone might have tried to coax Bush friendly evacuees to be first in line to be interviewed by ABC's cameras.   

MP hopes an enterprising reporter on scene will try to look into these questions.  It might teach all a lesson or two about the advantages of random sampling . . .  and maybe more . . .

Related Entries - Polls in the News

Posted by Mark Blumenthal on September 16, 2005 at 03:52 PM in Polls in the News | Permalink


Mark, you have any comments on the poll today that only 43% of evacuees want to return home? I saw a headline, but I've been swamped.

Posted by: Rick Brady | Sep 17, 2005 1:03:29 AM

I think it's safe to assume that ABC weren't looking for good news for this President in those interviews. If you want to make the case that the interviewees were secretly selected (or self-selected) to be Bush Pollyannas, it seems to me you have to explain why that happened on this occasion, and not on others.

Personally, I find it much easier to parse the situation in the same way one can in many others: if you ask people about (say) congressional overspending, most people will answer, sure, too much spending: but when you ask them about their own congressman, they're full of praise for him bringing home the pork, err, bacon.

And, of course, it goes without saying that the survey results are to some degree influenced by the media, which have systematically made President Bush and the administration out to be the bad guys, notwithstanding the very substantial evidence that Mayor Nagin and Governor Blanco were (to put it charitably) at least equally culpable and negligent.

Posted by: David Hecht | Sep 17, 2005 11:07:35 AM

...very risky for MP to boldly assert that the Washington Post/Kaiser/Harvard study:

["...was truly a "scientific" random sample survey of evacuees living in Houston area shelters"].


Certainly it was much more scientific than the ABC TV poll -- but as you know, 'true' random samples are difficult to get in practice.

For example, the Post/Kaiser/Harvard 'truly random' survey polled people at the five 'smaller' shelters at "various times" throughout the day; whereas, interviewers were at the large shelter continuously from 9:00 am to 8:00 pm to conduct interviews....
On what scientific basis then can we be sure that all members of the target survey population had an "equal" chance of interview selection ?

The target population was aged 18 and over --- how did the interviewers ensure no 17-year olds were counted?

Red Cross volunteers escorted the interviewers on some interviews; did the presence/absence of Red Cross personnel affect responses or response rate?

Privacy was attempted for interviews, given the various conditions & times.
Did the actual degree of privacy available for any specific interview affect things ?

Interviewers were given a random number and instructed to count
off this number of people before beginning the first/next interview....
but exactly how does one precisely count off the 'next' person in a jumbled & crowded shelter, with cots and people in every space corner, people moving about -- and not neatly aligned in numerical order ?

"Random" ain't so easy in the real world !

Posted by: Kelkins | Sep 18, 2005 4:00:45 AM

I am donating a house, you can help.

go here and find out how

Posted by: john | Sep 23, 2005 2:56:33 AM

The comments to this entry are closed.