A quick update on yesterday’s post: First, some of those who left comments yesterday seem to miss the main point. MP has no doubt that African-Americans express nearly monolithic disapproval of President Bush, far more than any other demographic subgroup. The question is whether the small sample of African Americans in this week’s NBC/Wall Street Journal poll justified the attention and “free fall” rhetoric it inspired.
MP contacted several public pollsters that also released national surveys in the last week. They were hesitant to release results from their small subsamples of of African-Americans because they rarely report on subgroups of less than 100 interviews. Here are two responses:
Kathy Frankovic of CBS News:
We have on a few occasions reported subgroups of less than 100 respondents. Usually that would be to make the point that there was nothing unusual about the group — or that there was something very different about them (blacks on whether the slow response to Hurricane Katrina was motivated by race, for example). But those are exceptions — usually we don’t even look when we know the sample size will be small.
Dana Blanton Stanton of Fox News:
When we get a request for a small subgroup, we try use characterizations rather than exact percentages. If pushed to use a number, we prefer to use a rolling average of three polls instead of just one poll
MP is sympathetic. The main point of yesterday’s item was to warn of the dangers (and questionable news value) of very small sample sizes. So pressuring the pollsters to release data from similarly small recent samples of African-Americans seems a bit hypocritical.
So rather than focus on their polls from last week, MP asked if they would be willing to roll together their recent samples to increase the sample sizes and tabulate the Bush job rating for two periods: Polls conduced since Katrina (from September to October) and those conducted earlier in 2005 (from January to August). Here are the results.
Some important notes: The CBS and Pew Research Center surveys — like the NBC/Wall Street Journal survey — interview national samples of adults. The Fox News/Opinion Dynamics conducts a survey of self-described registered voters only. Second, both CBS and Pew did additional “over-samples” of African Americans on their first post-Katrina studies in September. Also, the the number of interviews (“n=”) provided for the Pew and CBS surveys are unweighted totals. Finally, the omission of results for white voters by CBS is MP’s doing. He neglected to ask.
So what do these numbers tell us? With larger sample sizes, neither the CBS nor the Pew surveys show much change in Bush’s job approval among African-American voters since Katrina, although the view of Bush among black voters has obviously been very negative all along.
On the other hand, the Fox News survey of registered voters shows a big drop (from 22% to 7%) in Bush’s job rating among African-Americans. Whether this results from the difference between all adults and registered voters or some other factor is unclear.
In any event, these numbers strongly suggest that the 2% statistic understates Bush’s job rating among African Americans. The pollsters from Fox and CBS were willing to characterize their most recent samples as consistent with their recent polls conducted from September through October. So if we were to average the results among African Americans on all the polls released in the last week, the Bush job approval rating would probably be at about about 10%.
Why does this matter? Isn’t 10% a pretty low number? Yes, but consider it this way: If the NBC/WSJ poll had estimated Bush’s approval among African-Americans at 10% rather than 2%, would Tim Russert have gotten as excited (“2 percent!”) and reported the number as prominently on the Nightly News? Would the Washington Post’s Dan Froomkin have devoted a column to what “what may turn out to be one of the biggest free-falls in the history of presidential polling?” Would the Huffington Post have hyped the statistic with a banner front page headline? Would Maureen Dowd have included it in her (subscription required) column today?
MP doubts it.
And if you still find the 2% statistic “plausible,” ask yourself this question: If the next NBC/WSJ journal poll shows Bush’s approval rating at 10%, should Russert take to the airwaves to report a dramatic “rebound?”
LATE UPDATE: Looks like Russert continued to tout the 2% figure on Sunday’s Meet the Press (thanks to reader KW).
2% ain’t ever plausible (and I exaggerate only mildly).
2% flirts with the margins of unintentional erros of subject understanding of the question, subject matter recognition, response expression, interviewer perception, and response recording … not to mention the inevitably titre of deliberately non-serious responses and mentally incompetent subjects.
2% dares the extremes of the Fundamental Law of Social Science: “Some do and some don’t.”
And on the technical side, 2% gets you into territory where you’re not just exposed to small-sample effects, but error distributions that are more near-Poisson than near-Gaussian.
You can go broke betting either side of 2% propositions … and when you come out ahead, you may have a hard time taking your winnings to the bank.
I think we’re getting a little overly exercised over the 2% finding. I suspect most media commentators who talked about the 2% finding were just treating it as a curiosity, as something cute. It also makes for a funny punchline that, factoring in the margin of error, Bush’s approval among African-Americans could be as low as negative-1.
I would think — or at least, hope — that most of our political journalists appreciate the presence of small samples when looking at certain subgroups and the added fluctuations they entail.
And, as everyone acknowledges, Bush’s popularity among African-Americans is inordinately small in any case, whether 2% or 10%, so the 2% episode will likely not throw anyone’s perceptions seriously astray.
Mark has pursued this issue with his usual tenacity and thoroughness, from which I do not mean to detract. His idea to do an aggregate analysis of Bush’s approval ratings among African-Americans pre- and post-Katrina was very clever, in fact.
Who would trumpet a statistically irrelevant 2% stat that disses Bush? You named them: Tim Russert, former politica operative for a Democrat governer of NY; Brian Williams, former Democrat White House intern; Maureen Dowd, veteran NY Times foam-at-the-mouth Bush-loather; the predictably left-wing Washington Post; and HuffPo. Sadly, most people will only know about the poll from what these (and other) biased sources say about the poll.