Schiavo “Push Poll?”

Legacy blog posts Polls in the News

[3/24 – 2:45 p.m. EST –  posted additional updates below]

Another day, another polling controversy.  The latest involves a survey released on Monday by ABC News that shows 63 to 28 percent support for removal of Terry Schiavo’s feeding tube.  The survey drew intense interest in Washington and immediate allegations of biased question wording from the blogosphere’s right wing.  Captain’s Quarters called it a "push poll for euthanasia." Wizbang adds another adjective, calling it a "bogus push poll for euthanasia." 

Do they have a point?  The quick answer:  The evidence of bias or deliberate untruth in the ABC poll is scant, though the issue raises some interesting questions about the appropriateness of "informed" questions.

Now here’s the long version.

First, a plea for reporters, editors and bloggers of all ideologies:   Can we please stop using the term "push poll" to describe every survey we consider objectionable?  Yes, complain about bias when you see it, but the phrase push poll belongs to a higher order offense.  To summarize the definitions posted online by the American Association for Public Opinion Research (AAPOR), The National Council on Public Polls (NCPP) and the Council for Marketing & Opinion Research (CMOR):  A push poll is not a poll at all but rather a form of fraud – an effort to spread an untrue or salacious rumor under the guise of legitimate research.  "Push pollsters" are not pollsters at all.  They do not care about collecting data or measuring opinions (even in a "bogus" way).  They only care about calling as many people as possible to spread a false or malicious rumor without revealing their true intent.  Whatever complaint one might have about the wording or reporting of the ABC poll, it was certainly not a "push poll."

End rant.

Now to the more debatable question of whether the ABC poll was biased or unfair.  The complaints center mostly on the text of this question:

Schiavo suffered brain damage and has been on life support for 15 years. Doctors say she has no consciousness and her condition is irreversible. Her husband and her parents disagree about whether she would have wanted to be kept alive. Florida courts have sided with the husband and her feeding tube was removed on Friday. What’s your opinion on this case – do you support or oppose the decision to remove Schiavo’s feeding tube?

As noted above, 63% of the 501 adults surveyed on March 20 said they supported the decision, 28% opposed it and 9% had no opinion.  Sampling error was reported as  4.5%.

The main objection seems to be the use of the term "life support" in the second sentence. Again, from Captain’s Quarters:

Terri [Schiavo] has never been on life support. The only medical treatment Terri received for the past five years has been food and water through a feeding tube, which is nothing at all like artificial life support. Artificial life support consists of ventilation for people unable to breathe on their own. The question sets up a strawman argument that so completely contradicts reality that the entire poll must be considered invalid.

One test of this argument is a survey released by Gallup earlier this week (subscription only, also summarized here) conducted from Friday to Sunday, that asked a similar but more concise question without the use of the phrase "life support."

As you may know, on Friday the feeding tube keeping Terri Schiavo alive was removed. Based on what you have heard or read about the case, do you think that the feeding tube should or should not have been removed? 

Fifty-six percent (56%) of the 909 Gallup respondents said the tube should be removed, 31% said it should not be removed and 13% had no opinion.  Support for removing the tube is five points less than on the ABC poll, though the difference is not quite statistically significant. 

The Fox News Poll also asked the following "informed" question on a survey conducted March 1-2:

Terri Schiavo has been in a so-called ‘persistent vegetative state’ since 1990. Terri’s husband says his wife would rather die than be kept alive artificially and wants her feeding tube removed. Terri’s parents believe she could still recover and want the feeding tube to remain.  If you were Terri’s guardian, what would you do? Would you remove the feeding tube or would you keep the feeding tube inserted?

Fifty nine percent (59%) of Fox’s sample of 900 registered voters would remove the feeding tube, 24% would keep it inserted and 17% were unsure.   Note that the 35-point margin of support for removing Schiavo’s feeding tube is the same as on the ABC survey.

It is also worth noting that the ABC poll was completed in a single evening.  As the National Council on Public Polls (NCPP) points out: "Surveys conducted on one evening, or even over two days, have more sampling biases — due to non-response and non-availability — than surveys which are in the field for three, four or five days."   

Between the sampling error and the vagaries of one night samples, we cannot say conclusively that the ABC language produced more support for removing Schiavo’s feeding tube.  However, for the sake of argument, let’s concede that the ABC informed had such an effect.  Was the language of their question defensible?  [3-24 On some reflection a better word here would be "fair" – see comments below]

According to an article on the issue in yesterday’s New York Sun, ABC News Polling Director Gary Langer "said in an e-mail to the Sun that the descriptions were taken from an appellate court decision in Florida that described Mrs. Schiavo’s condition."   Here is one example from the Florida Supreme Court decision:

In this case, the undisputed facts show that the guardianship court authorized Michael to proceed with the discontinuance of Theresa’s life support after the issue was fully litigated in a proceeding in which the Schindlers were afforded the opportunity to present evidence on all issues. (p. 15 – emphasis added)

Moreover, the contention that the phrase "life support" in the ABC question automatically conjures up images of an artificial respirator rather than a feeding tube, thus creating a "strawman" that "completely contradicts reality" (as Captain’s Quarters put it) does not hold up.  In fact, the ABC question uses the phrase "feeding tube" twice, ultimately asking whether respondents support "the decision to remove Schiavo’s feeding tube."  In Cruzan v. Director, the U.S. Supreme Court held that tube feeding was legally no different from other forms of life support (see also this article).  Legality aside, it is hard to imagine that most respondents would interpret "the food fluids or medical treatment necessary to sustain her life" (the language in the law enacted by Congress over the weekend) as meaning something other than "life support." 

If the greater poll support for removing the feeding tube was more than random error, my hunch is that the effect had more to do with the statements that "Florida courts have sided with" the husband (they certainly did) and that "doctors say she has no consciousness and her condition is irreversible" (hard to quarrel with given reports like this one).   It might have been better to first ask a question that presented less information (as Gallup did), but calling the ABC description "untrue" or "deliberately slanted" is quite a stretch. 

This point leads to a more general objection that Rick Brady raised about the whole notion of "informed" questions:

Polling organizations like ABC News are not supposed to educate people regarding the issues they are polling. If a large portion of the public is not well informed on a subject matter related to an area in which they already have solidly formed opinions (I wouldn’t want to be on "life support" or a "vegetable," therefore I think Terri’s tube should be removed and Congress should stay out of it), in most cases, three sentences of preamble will not be sufficient to illicit a respondents true opinion.

I have to disagree with Rick, though I think it is fair to say he speaks for a vocal minority of academic survey methodologists.  Political pollsters frequently encounter complex issues about which the public lacks knowledge or "solidly formed opinions."  The Schiavo case is a perfect example.  Even after the blanket coverage of last weekend, nearly half of ABC’s respondents said they had been following the case "not very closely" (16%) or "not at all" (28%). 

As a result, we frequently ask questions that first provide a bit of information or context, especially when an issue is poised to get much greater attention or become the focus of a political campaign.  I have written hundreds, perhaps thousands of such questions, and 99% of the time the results are not intended for public consumption.  Our goal is not to create propaganda but to accurately gauge how opinions might develop with more information, and we struggle to find language that simulates the dialogue that will ultimately play out in the media.  As the Schiavo example proves, this task is not easy. 

Republican pollster John McLaughlin told the NY Sun that he "would have worded the [Schiavo] questions differently."  I am sure that’s true – I probably would have taken a different tack as well.   However, as with recent "informed" questions on Social Security, if you give this task to 20 pollsters, you will likely get 20 different questions.  It is easy to quibble with ABC’s approach but the charge from Captain’s Quarters that they were either "incompetent" or "attempted to fool their viewers and readership with false polling that essentially lies about the [Schiavo] case" is grossly unfair.

————

UPDATE:  Gallup released a subsequent one-night poll this morning. Among the questions asked:

As you may know, Terri Schiavo is a Florida woman in a persistent vegetative state who was being kept alive through the use of a feeding tube. The feeding tube was removed on Friday, an action that will result in her death within about two weeks. A federal judge made a ruling in the case today.  First: Do you agree with the federal judge’s decision that resulted in the feeding tube being left unattached, or do you disagree and think the federal judge should have ordered the feeding tube to be re-attached?"

52% agree with judge
39% disagree with judge
9% unsure

———-
UPDATE II:  Our friend Mickey Kaus disagrees, to put it mildly. The crux of his argument is the notion that the public perceives "life support" to mean a respirator and a patient who will stop breathing within minutes of its removal, a condition they argue is considerably worse than Jennifer Terri Schiavo’s.  If that were true, and if the public perceived it that way, the term "life support" would likely bias the results.  I am asking if we have any evidence of such a perception. 

The best place to look is the other polls that make no reference to "life support" or what "doctors say" about Schaivo’s state of consciousness or chances of survival.  Fox showed 59% support for removing the tube, Gallup showed 58% support.    Thus,  Mickey points out that "no other poll has as large an anti-tube majority (63%) as ABC’s." 

That was true until CBS released a new survey tonight (text from the full pdf via RealClearPolitics).  Here are the key questions:

Q13. Terri Schiavo has been in a persistent vegetative state since 1990. Terri’s husband says his wife would not want to be kept alive under these circumstances and he wants her feeding tube removed. Terri’s parents believe her condition could improve and they want the feeding tube to remain. How closely have you been following news about the case — have you been following it very closely, somewhat closely, not too closely, or not at all?

32% very closely

44% somewhat closely

17% not too closely
6% not at all

1% don’t know

***Partial Sample***
Q14. What do you think should have happened in this case — should the feeding tube have been removed or should it have remained?
61% Should be removed
28% Should remain
       
11% Don’t know

Q19 What should happen now? Should the feeding tube be re-inserted, or not?
       
27% Reinsert
66% Not
7% Don’t know

These questions appear to indicate an "anti-tube" majority of roughly the same size as indicated by the ABC poll without any mention of "life support" or what "doctors say" about Schiavo’s condition.

However, I’ll hedge for the time being, only because the CBS release is a bit confusing.  First, the heading "Partial Sample" appears over the results for Q14 appears to also apply to Q19.  This would usually imply that the sample of 737 adults was randomly divided in two, with half the respondents hearing Q14 and half Q19.  That would normally make sense, although asking “what should happen now” in Q19 without the introduction from Q14 seems odd.  Adding to my confusion is the jump in the labeling from Q14 to Q19, followed by Q15 through Q18.  Hopefully, someone from CBS will help clarify the mechanics.

UPDATE III:   I spoke with Kathy Frankovic at CBS who helped clarify their release.  The two questions on whether the Schiavo feeding tube should have been removed or reattached were labeled "partial sample" because they were only asked on the second night of interviewing.  The survey had originally included a different version of Q14 that had been written erroneously in the future tense (e.g. What do you think should happen in this case — should the feeding tube be removed or should it remain?).  Since that language was inaccurate (the tube had already been removed), they decided to replace that question with the two-question sequence above for the second night of interviewing.

As a result, these two questions (Q14 & Q19) were asked of fewer respondents (n=321 unweighted) than the full sample (n=737) and thus had a bigger margin of error (+/-6%) than the full sample (+/- 4%) .   Also,  results for these two questions, like the survey done by ABC earlier in the week, are subject to the same caveats about one night polls described above. 

According to Frankovic, the second new question was labeled Q19 because they typically leave gaps in question numbering for insertions of new questions when these situations arise.  The questions were asked in the order the appear in the relese.    Note that the PDF document has no Q7, Q11 or Q12.

Frankovic also provided the party identification results for the weighted data:  27% Republican, 32% Democrat, 41% other or don’t know.    Some may have seen different numbers posted in error today on DailyKos.

ONE LAST THOUGHT:  After reflecting on the comments on this post, there is one word I wish I had written differently: “defensible” (as in, “was the language of their question defensible?”).  A better word would have been “fair” or as Kaus put it, “reasonably calculated to produce an accurate poll of what people think.” 

Gerry Daly, who also had problems with my use of “defensible,” wrote: “We should strive for them to conduct polls that have fair wording and that provide the most bias-free read of the public that is reasonably attainable.”  No argument there. 

My point about the court documents and the legal definition stemming from the Cruzan decision was not just about technical definitions but about language:   This is a matter of opinion (for now), but I doubt that ordinary Americans are making the same distinctions regarding “life support” and “tube feeding” as those who are passionately interested in this story.  Remove the feeding tube and Terri Schiavo dies.  As a matter of language and plain meaning, how is that not “life support?”   

So in that regard, I think that while far from perfect, the ABC question was fair.   Others — obviously — disagree.   Read the comments for a sampling.

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.