March 25, 2005
Schiavo: The Return of Professor M
I thought that in light of the interest in the last post on the recent Schiavo polls that it would be good to take a step back from the microanalysis of and write generally about how pollsters write questions about issues and public policy. I was pleasantly surprised to find that our old friend Professor M, a member of the Political Science faculty at a small midwestern college, had posted some comments that accomplished much of this task for me, and said it better than I would have. For those who do not browse the comments section, his comments are more than worthy of promotion to the main page. This is today's must read:
Mark, I think that your discussion here implicitly endorses a commonly held error about the best way to interpret polling data about matters of public interest. (And this error underlies the criticism of the ABC poll as well.)
The error is the incorrect belief that there is a "right" or "unbiased" way to ask a question about any given public issue. There is no such thing. Everyone who works within the polling field is well aware that small changes in wording can affect the ways in which respondents answer questions. This approach leads us into tortuous discussions of question wording on which reasonable people can differ. Further, as you have pointed out many times in the past, random variation in the construction of the sample or in response rates can skew the results of any single poll away from the true distribution of opinions in the population.
So how do we look at public opinion on an issue such as the Schiavo case? The answer is NOT to find a single poll with the "best" wording and point to its results as the final word on the subject. Instead, we should look at ALL of the polls conducted on the issue by various different polling organizations. Each scientifically fielded poll presents us with useful information. By comparing the different responses to multiple polls -- each with different wording -- we end up with a far more nuanced picture of where public opinion stands on a particular issue. If we can see through such comparisons that stressing different arguments or pieces of information produces shifts in responses, then we have perhaps learned something. Like our own personal opinions, public opinion is not some sort of simple yes/no set of answers; it is complex, and it can see both sides of complicated issues when presented with enough information.
If we were to lock pollsters of all partisan persuasions in a room and force them to pick the "best" question wording on the Schiavo issue, we might end up with everyone asking the same question, but overall we would end up with less information about public opinion, not more. We are better off having the wide variety of different polls, with questions stressing different points of view on the issues, and then comparing them all to one another. This is precisely what you do in your discussion of the ABC poll, but I think you are asking entirely the wrong question -- not "is the ABC wording defensible?" but rather "what does the ABC poll, when compared to other polls with different wording, add to our overall understanding of public opinion on this issue?"
Of course, this sort of contextualizing of polling results is exceedingly rare in the media. Much more common is the front page story saying "here is our poll, and here is what it found, and it is a true representation of public opinion" -- and by implication, no other poll matters. Intellectual honesty is trumped by competition. The best we usually get are vague generalizations of all of the polls lumped together ("polls have consistently shown disapproval of Congress' actions"), and even those generalizations almost never appear in the initial story trumpeting the "exclusive" poll fielded by the newspaper/network itself.
The end result is that even those who pay close attention to the news media and the chattering classes often have very little real understanding of how to interpret polls in a thoughtful way -- which is one of the reasons your blog is so valuable.
P.S. Polls which attempt to predict election results are a rather different kettle of fish, for two important reasons: (1) Pollsters have been experimenting with questions wording for over 50 years and can keep wording the same regardless of the issues in a race; and (2) There is an actual real-world "check" on pollsters' work in the form of the actual election results. Neither of these characteristics apply to polling about issues of public interest.
For those who want to look at all the recent polls on the Schiavo case, the PollingReport provides a great compilation that includes complete wording, sample sizes, interview dates and margins of error.
I've got some additional thoughts...but it's late. More on this topic tomorrow.
Related Entries - Polls in the News
Yup. This is why I found the following paragraph from your previous post somewhat wrong:
"ONE LAST THOUGHT: After reflecting on the comments on this post, there is one word I wish I had written differently: “defensible” (as in, “was the language of their question defensible?”). A better word would have been “fair” or as Kaus put it, “reasonably calculated to produce an accurate poll of what people think.”
"Defensible" was the right word originally. Polls are not supposed to be "fair". They're supposed to help elicit understanding.
Leave it to others to worry about mau-mau-ing the media to get favorable coverage of favorable poll results. Stick to helping to elicit understanding.
Posted by: Petey | Mar 25, 2005 3:20:47 AM
Professor M writes, "I think that you are asking entirely the wrong question -- not 'is the ABC wording defensible?' but rather 'what does the ABC poll, when compared to other polls with different wording, add to our overall understanding of public opinion on this issue?'" Far from thinking your question is entirely wrong, I think both questions are important.
If, as critics have argued, ABC's question actually misled respondents about the facts of the Schiavo case, it would still contribute to our overall understanding, but it would be indefensible to the extent that it was misleading. Or perhaps I should say that reporting the result in the lead would be indefensible. Suppose that ABC asked a question that began with the premise that "Social Security is going bankrupt within 40 years," and then headlined the result without further comment. That would again contribute to our knowledge of public opinion (although it may well violate AAPOR standards to ask such a question), but the reporting would be beyond the pale.
I agree with Petey that "defensible" was better than "fair." As Professor M argues in detail, there is no one accurate or right or fair way of posing the question. But precisely because the media do not and will not do a very good job of contextualizing survey findings, I don't think we can dismiss the defensibility question.
Posted by: Mark Lindeman | Mar 25, 2005 7:35:20 AM
Dr. Lindeman makes a good point. I perhaps overstated the case by implying that decent or "fair" question wording is totally unimportant. However, to suggest that we must accept the inability of media outlets to properly interpret their own data -- as you seem to do in the last sentence -- is IMHO giving up without a fight.
One of the interesting things about the blog phenomenon is the appearance of a much more robust criticism of the news media than was previously available in wide distribution (and media decision-makers even seems to notice and adjust their behavior occasionally). If through hearing such criticism, news reporters actually manage to find their way to better reporting and greater contextualization in the use of media-based polling, the defensibility question would become less important.
Posted by: Professor M | Mar 25, 2005 6:03:38 PM
Yeah I said something similar to Professor M, albeit less eloquently:
"More theoretically, any poll result is inherently tied to question wording, question order, response choice, moment in time, etc. so you can then say that asking any question with any set of answers and any order is by definition presenting a 'biased view.'"
Posted by: Chris Kennedy | Mar 25, 2005 10:12:18 PM
Have there been any polls that simply asked people whether they think the tube should have been removed without giving them *any* background informaton whatever? In a sense, this is the "fairest" solution, since *any* background information can be denounced as selective. (To be sure, such polls would exclude people who had heard nothing at all about the case, but by now you's have to have been living in a cave for the past two weeks for that to occur.)
Posted by: David T | Mar 26, 2005 2:36:18 PM
David, somebody HAS taken a poll with that question asked, and without a lot of "context". And the results indicate public opinion is the exact opposite of what one would conclude from the Gallup et al. polls. A John Zogby poll asked the bare question of whether a disabled person like Schaivo "should not have food and water denied". Only 9% said that the food/water should be cut off. 79% thought it shouldn't. An article on the poll is at www.lifenews.com/bio891.html, and a link to it is in Mickey Kaus's website kausfiles that our genial host often cites.
Now, I don't hold Zogby as one of America's more reliable pollsters (for example, just before the election he guaranteed a Kerry victory). However, the results are there. The margin is so great that even a few points of poll sloppiness couldn't effect the overall tendency. Makes one wonder why Gallup et al. aren't asking questions like this.
It goes to reinforce the thesis of Professor "M"--since question phrasing has inherent bias, one should consult a wide range of polls and a wide range of possible phrasings of questions.
Posted by: CivilWarGuy | Apr 4, 2005 11:22:14 AM
Thank you for great web site, i have discovered it recently, lots of good read
Posted by: stephanie | Apr 4, 2005 2:28:02 PM
CivilWarGuy: "should not have food and water denied" and "should the feeding tube have been removed" would not necessarily get the same result. It seems to me that the second one is more precise.
Posted by: David T | Apr 5, 2005 9:51:34 PM
your blog is cool, never stop blogging.
Posted by: natalee | May 14, 2006 1:27:02 AM
ok, i know, thanks, share your thoughts for us over and over.
Posted by: real personal chatting blog for finding friends | May 31, 2006 6:24:12 AM
The comments to this entry are closed.