April 21, 2006
The Question That Answers Itself
A few weeks ago, our friend Mickey Kaus described a question asked on a recent Time Magazine poll as having "comically biased wording." I was not ready to be quite so harsh about that particular poll. Well, this week courtesy of Carl Bialik of the Wall Street Journal Online, we have a different poll conducted by Zogby International whose questions and their wording truly meet the "comical" standard.
The second half of Bialik's weekly Numbers Guy column looks at a recent Zogby survey on online gambling sponsored by the online gambling industry. "It appears that the sponsor of the poll influenced the way it was conducted," Bialike writes, "particularly in the way the questions were phrased." He is putting it mildly.
Here is the most brazen of the questions used in the survey press release to support the assertion that "Americans overwhelmingly do not want" federal laws restricting online gambling:
More than 80% of Americans believe that gambling is a question of personal choice that should not be interfered with by the government. Do you agree or disagree that the federal government should stop adult Americans from gambling with licensed and regulated online sports books and casinos based in other countries?
Yes, you're reading that right. The text of the Zogby question actually answers itself. Or, to be more precise, it tells the respondent what "80% of Americans believe" about government regulation of gambling just before asking them what they believe about such regulation. It is thus not exactly surprising, as Bialik put it,
that after being told that most Americans don't want the government to interfere, some 71% of the respondents to this question signaled they, too, were against a government ban.
To be serious for a moment, the issue here is that the poll press release makes the following claim:
[The poll] establishes that Americans overwhelmingly do not want the federal government enacting laws that restrict a recreational activity such as online gambling.
No. At best, these results establish that a pollster can push respondents to oppose such restrictions. The obviously leading nature of the questions cited in the release makes them of little value in measuring the opinions Americans currently hold about online gambling. It is one thing to design "projective" questions in order to "see how different arguments play" (as Humphrey Taylor of Harris Interactive puts it in the Bialik article). It is quite another to try to pass off such projective questions as a "fair and balanced" reading on what Americans currently think, which is exactly what this press release does.
There is much more in Bialik's piece, including reaction from AAPOR President Cliff Zukin and a response from Zogby spokesman Fritz Wenzel. It is definitely worth reading in full.
However, MP has a hunch there is more to this story.
I am doing some additional digging, but here's a hint: The press release describes the study as a "scientific poll of over 30,000 likely voters" interviewed over a two week period with a "margin of error" of " 0.6 percentage points." Moreover, according to Bialik's column, the survey sponsor claims they paid "less than $10,000" for the survey.
I'll put it this way: I'm aware of no pollster or calling center that will complete a telephone survey of 30,000 likely voters for less than 33 cents an interview.
More to come...
What's your opinion of this?
Posted by: Robert Chung | Apr 21, 2006 3:03:15 PM
Some of the questions that Robert points to (from the most recent Fox News poll) are one-sided and leading. They are also "projective" (as that term was used above) in that they collectively present a one-sided argument about the job the news media does covering the nation's economy before probing opinions about the news media. They are testing how Americans react to an argument, not what Americans believe.
The author of the questions obviously want to argue that Americans are poorly informed about the economy, and that when better informed (from the author's perspective), they will condemn news media coverage of the economy.
I would be critical of Fox News if they present the results of Question #37 (the one at the end of the series on the job the news media is doing) as a measurement of what Americans believe *now.* If they present the results as a guage of how Americans *might react* to an argument, and they provide the full text of that argument, I would be far less critical. Other pollsters may disagree, but it seems reasonable to use polling to measure how the public reacts to political messages or arguments. Campaign pollsters like me do that all that time.
One thing that would have made these questions far more informative would have been at an "unaided" item to come first without any introduction: "Do you think the news media has generally done a good job or bad job providing accurate news about the nation’s economy?" Then we could see how many of those who answer Q37 by saying the media does a "bad job" covering the economy held that opinion *before* hearing all the leading information presented in Q32, Q34, Q36 and Q37.
Another distinction with the Zogby poll (as presented) is that that the Fox poll includes some non-leading probes of how well (or poorly) Americans are informed about some of these issues (Q31, Q33 & Q35). These are useful in evaluating the underlying hypothesis being tested.
Posted by: Mark Blumenthal | Apr 22, 2006 10:20:44 AM
I actually took this one, online at Zogby via an e-mailed invitation.
I get them all the time, probably one every other week or so. This one is definitely the most eye-roll-inducing one ever -- the vast majority of them are standard opinion polls on Bush approval, etc.
Posted by: Ken Alper | Apr 22, 2006 5:50:48 PM
You may find it interesting that the Poker Players Alliance commissioned a study that my company (ICR) conducted. The results of their study (yes, a randomly sampled telephone survey) are the product of what I feel are unbiased, non-leading questions. With the exception of the last question in the survey, though, the topic is "poker" -- a specific and culturally-popular form of gambling.
Posted by: Gregory Kohs | Apr 23, 2006 1:39:38 PM
The comments to this entry are closed.