« Tired of Exit Polls Yet? | Main | Schiavo: The Return of Professor M »

March 23, 2005

Schiavo "Push Poll?"

[3/24 - 2:45 p.m. EST -  posted additional updates below]

Another day, another polling controversy.  The latest involves a survey released on Monday by ABC News that shows 63 to 28 percent support for removal of Terry Schiavo's feeding tube.  The survey drew intense interest in Washington and immediate allegations of biased question wording from the blogosphere's right wing.  Captain's Quarters called it a "push poll for euthanasia." Wizbang adds another adjective, calling it a "bogus push poll for euthanasia." 

Do they have a point?  The quick answer:  The evidence of bias or deliberate untruth in the ABC poll is scant, though the issue raises some interesting questions about the appropriateness of "informed" questions.

Now here's the long version.

First, a plea for reporters, editors and bloggers of all ideologies:   Can we please stop using the term "push poll" to describe every survey we consider objectionable?  Yes, complain about bias when you see it, but the phrase push poll belongs to a higher order offense.  To summarize the definitions posted online by the American Association for Public Opinion Research (AAPOR), The National Council on Public Polls (NCPP) and the Council for Marketing & Opinion Research (CMOR):  A push poll is not a poll at all but rather a form of fraud - an effort to spread an untrue or salacious rumor under the guise of legitimate research.  "Push pollsters" are not pollsters at all.  They do not care about collecting data or measuring opinions (even in a "bogus" way).  They only care about calling as many people as possible to spread a false or malicious rumor without revealing their true intent.  Whatever complaint one might have about the wording or reporting of the ABC poll, it was certainly not a "push poll."

End rant.

Now to the more debatable question of whether the ABC poll was biased or unfair.  The complaints center mostly on the text of this question:

Schiavo suffered brain damage and has been on life support for 15 years. Doctors say she has no consciousness and her condition is irreversible. Her husband and her parents disagree about whether she would have wanted to be kept alive. Florida courts have sided with the husband and her feeding tube was removed on Friday. What's your opinion on this case - do you support or oppose the decision to remove Schiavo's feeding tube?

As noted above, 63% of the 501 adults surveyed on March 20 said they supported the decision, 28% opposed it and 9% had no opinion.  Sampling error was reported as  4.5%.

The main objection seems to be the use of the term "life support" in the second sentence. Again, from Captain's Quarters:

Terri [Schiavo] has never been on life support. The only medical treatment Terri received for the past five years has been food and water through a feeding tube, which is nothing at all like artificial life support. Artificial life support consists of ventilation for people unable to breathe on their own. The question sets up a strawman argument that so completely contradicts reality that the entire poll must be considered invalid.

One test of this argument is a survey released by Gallup earlier this week (subscription only, also summarized here) conducted from Friday to Sunday, that asked a similar but more concise question without the use of the phrase "life support."

As you may know, on Friday the feeding tube keeping Terri Schiavo alive was removed. Based on what you have heard or read about the case, do you think that the feeding tube should or should not have been removed? 

Fifty-six percent (56%) of the 909 Gallup respondents said the tube should be removed, 31% said it should not be removed and 13% had no opinion.  Support for removing the tube is five points less than on the ABC poll, though the difference is not quite statistically significant. 

The Fox News Poll also asked the following "informed" question on a survey conducted March 1-2:

Terri Schiavo has been in a so-called 'persistent vegetative state' since 1990. Terri's husband says his wife would rather die than be kept alive artificially and wants her feeding tube removed. Terri's parents believe she could still recover and want the feeding tube to remain.  If you were Terri's guardian, what would you do? Would you remove the feeding tube or would you keep the feeding tube inserted?

Fifty nine percent (59%) of Fox's sample of 900 registered voters would remove the feeding tube, 24% would keep it inserted and 17% were unsure.   Note that the 35-point margin of support for removing Schiavo's feeding tube is the same as on the ABC survey.

It is also worth noting that the ABC poll was completed in a single evening.  As the National Council on Public Polls (NCPP) points out: "Surveys conducted on one evening, or even over two days, have more sampling biases -- due to non-response and non-availability -- than surveys which are in the field for three, four or five days."   

Between the sampling error and the vagaries of one night samples, we cannot say conclusively that the ABC language produced more support for removing Schiavo's feeding tube.  However, for the sake of argument, let's concede that the ABC informed had such an effect.  Was the language of their question defensible?  [3-24 On some reflection a better word here would be "fair" - see comments below]

According to an article on the issue in yesterday's New York Sun, ABC News Polling Director Gary Langer "said in an e-mail to the Sun that the descriptions were taken from an appellate court decision in Florida that described Mrs. Schiavo's condition."   Here is one example from the Florida Supreme Court decision:

In this case, the undisputed facts show that the guardianship court authorized Michael to proceed with the discontinuance of Theresa's life support after the issue was fully litigated in a proceeding in which the Schindlers were afforded the opportunity to present evidence on all issues. (p. 15 - emphasis added)

Moreover, the contention that the phrase "life support" in the ABC question automatically conjures up images of an artificial respirator rather than a feeding tube, thus creating a "strawman" that "completely contradicts reality" (as Captain's Quarters put it) does not hold up.  In fact, the ABC question uses the phrase "feeding tube" twice, ultimately asking whether respondents support "the decision to remove Schiavo's feeding tube."  In Cruzan v. Director, the U.S. Supreme Court held that tube feeding was legally no different from other forms of life support (see also this article).  Legality aside, it is hard to imagine that most respondents would interpret "the food fluids or medical treatment necessary to sustain her life" (the language in the law enacted by Congress over the weekend) as meaning something other than "life support." 

If the greater poll support for removing the feeding tube was more than random error, my hunch is that the effect had more to do with the statements that "Florida courts have sided with" the husband (they certainly did) and that "doctors say she has no consciousness and her condition is irreversible" (hard to quarrel with given reports like this one).   It might have been better to first ask a question that presented less information (as Gallup did), but calling the ABC description "untrue" or "deliberately slanted" is quite a stretch. 

This point leads to a more general objection that Rick Brady raised about the whole notion of "informed" questions:

Polling organizations like ABC News are not supposed to educate people regarding the issues they are polling. If a large portion of the public is not well informed on a subject matter related to an area in which they already have solidly formed opinions (I wouldn't want to be on "life support" or a "vegetable," therefore I think Terri's tube should be removed and Congress should stay out of it), in most cases, three sentences of preamble will not be sufficient to illicit a respondents true opinion.

I have to disagree with Rick, though I think it is fair to say he speaks for a vocal minority of academic survey methodologists.  Political pollsters frequently encounter complex issues about which the public lacks knowledge or "solidly formed opinions."  The Schiavo case is a perfect example.  Even after the blanket coverage of last weekend, nearly half of ABC's respondents said they had been following the case "not very closely" (16%) or "not at all" (28%). 

As a result, we frequently ask questions that first provide a bit of information or context, especially when an issue is poised to get much greater attention or become the focus of a political campaign.  I have written hundreds, perhaps thousands of such questions, and 99% of the time the results are not intended for public consumption.  Our goal is not to create propaganda but to accurately gauge how opinions might develop with more information, and we struggle to find language that simulates the dialogue that will ultimately play out in the media.  As the Schiavo example proves, this task is not easy. 

Republican pollster John McLaughlin told the NY Sun that he "would have worded the [Schiavo] questions differently."  I am sure that's true - I probably would have taken a different tack as well.   However, as with recent "informed" questions on Social Security, if you give this task to 20 pollsters, you will likely get 20 different questions.  It is easy to quibble with ABC's approach but the charge from Captain's Quarters that they were either "incompetent" or "attempted to fool their viewers and readership with false polling that essentially lies about the [Schiavo] case" is grossly unfair.

------------

UPDATE:  Gallup released a subsequent one-night poll this morning. Among the questions asked:

As you may know, Terri Schiavo is a Florida woman in a persistent vegetative state who was being kept alive through the use of a feeding tube. The feeding tube was removed on Friday, an action that will result in her death within about two weeks. A federal judge made a ruling in the case today.  First: Do you agree with the federal judge's decision that resulted in the feeding tube being left unattached, or do you disagree and think the federal judge should have ordered the feeding tube to be re-attached?"

52% agree with judge
39% disagree with judge
9% unsure

----------
UPDATE II:  Our friend Mickey Kaus disagrees, to put it mildly. The crux of his argument is the notion that the public perceives "life support" to mean a respirator and a patient who will stop breathing within minutes of its removal, a condition they argue is considerably worse than Jennifer Terri Schiavo's.  If that were true, and if the public perceived it that way, the term "life support" would likely bias the results.  I am asking if we have any evidence of such a perception. 

The best place to look is the other polls that make no reference to "life support" or what "doctors say" about Schaivo's state of consciousness or chances of survival.  Fox showed 59% support for removing the tube, Gallup showed 58% support.    Thus,  Mickey points out that "no other poll has as large an anti-tube majority (63%) as ABC's." 

That was true until CBS released a new survey tonight (text from the full pdf via RealClearPolitics).  Here are the key questions:

Q13. Terri Schiavo has been in a persistent vegetative state since 1990. Terri's husband says his wife would not want to be kept alive under these circumstances and he wants her feeding tube removed. Terri's parents believe her condition could improve and they want the feeding tube to remain. How closely have you been following news about the case -- have you been following it very closely, somewhat closely, not too closely, or not at all?
32% very closely
44% somewhat closely
17% not too closely
6% not at all
1% don't know

***Partial Sample***
Q14. What do you think should have happened in this case -- should the feeding tube have been removed or should it have remained?
61% Should be removed
28% Should remain        
11% Don't know

Q19 What should happen now? Should the feeding tube be re-inserted, or not?        
27% Reinsert
66% Not
7% Don't know

These questions appear to indicate an "anti-tube" majority of roughly the same size as indicated by the ABC poll without any mention of "life support" or what "doctors say" about Schiavo's condition.

However, I’ll hedge for the time being, only because the CBS release is a bit confusing.  First, the heading "Partial Sample" appears over the results for Q14 appears to also apply to Q19.  This would usually imply that the sample of 737 adults was randomly divided in two, with half the respondents hearing Q14 and half Q19.  That would normally make sense, although asking “what should happen now” in Q19 without the introduction from Q14 seems odd.  Adding to my confusion is the jump in the labeling from Q14 to Q19, followed by Q15 through Q18.  Hopefully, someone from CBS will help clarify the mechanics.

UPDATE III:   I spoke with Kathy Frankovic at CBS who helped clarify their release.  The two questions on whether the Schiavo feeding tube should have been removed or reattached were labeled "partial sample" because they were only asked on the second night of interviewing.  The survey had originally included a different version of Q14 that had been written erroneously in the future tense (e.g. What do you think should happen in this case -- should the feeding tube be removed or should it remain?).  Since that language was inaccurate (the tube had already been removed), they decided to replace that question with the two-question sequence above for the second night of interviewing.

As a result, these two questions (Q14 & Q19) were asked of fewer respondents (n=321 unweighted) than the full sample (n=737) and thus had a bigger margin of error (+/-6%) than the full sample (+/- 4%) .   Also,  results for these two questions, like the survey done by ABC earlier in the week, are subject to the same caveats about one night polls described above. 

According to Frankovic, the second new question was labeled Q19 because they typically leave gaps in question numbering for insertions of new questions when these situations arise.  The questions were asked in the order the appear in the relese.    Note that the PDF document has no Q7, Q11 or Q12.

Frankovic also provided the party identification results for the weighted data:  27% Republican, 32% Democrat, 41% other or don't know.    Some may have seen different numbers posted in error today on DailyKos.

ONE LAST THOUGHT:  After reflecting on the comments on this post, there is one word I wish I had written differently: “defensible” (as in, “was the language of their question defensible?”).  A better word would have been “fair” or as Kaus put it, “reasonably calculated to produce an accurate poll of what people think.” 

Gerry Daly, who also had problems with my use of “defensible,” wrote: “We should strive for them to conduct polls that have fair wording and that provide the most bias-free read of the public that is reasonably attainable.”  No argument there. 

My point about the court documents and the legal definition stemming from the Cruzan decision was not just about technical definitions but about language:   This is a matter of opinion (for now), but I doubt that ordinary Americans are making the same distinctions regarding “life support” and “tube feeding” as those who are passionately interested in this story.  Remove the feeding tube and Terri Schiavo dies.  As a matter of language and plain meaning, how is that not “life support?”   

So in that regard, I think that while far from perfect, the ABC question was fair.   Others -- obviously -- disagree.   Read the comments for a sampling.

Related Entries - Polls in the News

Posted by Mark Blumenthal on March 23, 2005 at 02:23 PM in Polls in the News | Permalink

Comments

Good post Mark. As I made clear in my post, I didn't think the ABC poll was a Push Poll. You're the one with the experience designing survey instruments and I should have qualified my comments on what polling orgs are or are not "supposed" to do. :-)

Sentences of my post following your quote read: "I'm sure there are those who viewed the questions asked by ABC News as pushing respondents into pro-Schiavo and pro-Congress answers. That's what makes the design of survey instruments so difficult."

I only meant to suggest that polling organizations have a tough time providing enough accurate and unbiased information about complex or controversial issues in a few sentences to elicit reliable responses (there's that whole reliability testing issue one of your readers educated me on a while back).

Regarding the accuracy of terms like "persistent vegetative state (PVS)" and "life support" in describing Terri's state, I think that is open to debate, and hence the unwarranted charges of "Push Poll" from the right.

By telling poll respondents that Terri is PVS, I feel that ABC and Gallup violated a principle of good survey design that my professor calls "level of wording." PVS is a technical medical term and using PVS in question presumes that people understand that term.

My hypothesis is that when people hear "persistant vegetative state" they think "brain dead vegetable." Terri is by no means brain dead and I would argue that she is not a "vegetable" either. Brain dead vegetables on life support have no chance of recovery.

I've heard and read much testimony supporting both sides of this argument - including from people who have been declared PVS by doctors, but fully recovered and are now on the talk shows telling their story (not to imply that recovery from PVS is common, or that Terri can recover [I simply don't know]). PVS does not mean brain dead vegetable with no chance of recovery. But my bet is that is what the public thinks of when they hear the term "persistent vegetative state."

Perhaps the polls should have said that Terri was "severely brain damaged" and that her "court appointed doctors" do not believe that she can improve (both accurate and fair statements at an appropriate level of wording).

As I said in my post about the "life support" versus "feeding tube" terms, use of these terms (PVS and feeding tube) may have "pushed" SOME respondents into answers, but I can't imagine that it would realign public opinion.

These are very interesting questions, and I must say that I am firmly in your camp. The ABC News poll was NOT a Push
Poll as that term should be reserved for acts of deviousness. But, I do think that a poll can "push" respondents into answers, and not be a Push Poll. That is why we can debate whether a poll question is biased without calling it a Push Poll.

Unfortunately though this is a nuance that most pundits do not appreciate. You provide the blogosphere (and media) a valuable service with this education.

Posted by: Rick Brady | Mar 23, 2005 4:11:03 PM

Mark, I don't think it was a push poll, but it does suggest a liberal bias on behalf of ABC pollsters. They subconsciously (my guess) labeled it "life support" because that is what most liberals believe. It is the Bernie Goldberg theory. Anyone who believed that that Terry should live would not have used that language.

Posted by: Cableguy | Mar 23, 2005 4:41:33 PM

Kaus is hacktackular as usual on this subject...

Posted by: Petey | Mar 23, 2005 6:11:03 PM

Sir:

In perhaps an unkind turn of a phrase, I have "no dog in this race." As I wrote on my blog SayetRight.Blogspot.Com I am torn. Not an easy position to take when one is pro-life for the most part and rather conservative after a lifetime of being a liberal.

Being torn, though, does not mean being blind and to categorize the dishonest polling of ABC News as anything other than a "push poll" is wrong.

This was nothing other than the attempt to spread misinformation and disinformation to the public (and then use the results to engender greater support for their position) by the same network that sent memos around during the campaign ordering their reporters not to offer the same degree of scrutiny to John Kerry as they are to offer George Bush.

As I point out in other posts, if these things were just "accidents" there would be a degree of randomness to them. Instead the "bad choice of words" and the forged documents and the lies about the streets of Iraq being too chaotic to allow for voting ALWAYS benefit the leftist agenda.

Posted by: Evan Sayet | Mar 23, 2005 8:13:34 PM

A few thoughts on the poll:

1. Any summary of "the facts of the case" in a few sentences must be intellectually inadequate, since most cases (including this one) cannot be summarized fairly in a few sentences. Thus any poll question that is prefaced by such a summary is suspect.
2. The claim that the summary would make Jack Webb proud ("Just the facts, ma'am") is also intellectually inadequate. One can write a summary using undeniably true, or arguably true, statements, and by cherrypicking the facts you use, present a biased view of the case.
3. The poll question response is thus inevitably biased by any prefatory summary.
4. The people who use the term "push poll" here are no more (or less) limited to the technical definition of "push poll" than other people are limited to the technical definition of "life support".

Posted by: CivilWarGuy | Mar 23, 2005 11:28:38 PM

Suppose the poll question had been worded thusly: "Terri Schiavo has been in a brain damaged state for a number of years. Michael Schiavo, Terri's husband, wishes to kill her by starving her to death. Her parents wish to keep her alive by feeding her. Do you support starving her to death or keeping her alive?"

Phrased this way, I suspect you'd get a much different poll response than the ABC poll. And note that everything said in the "context portion" is true--certainly as "true" as the ABC poll's context portion.

Posted by: Bruce Allardice | Mar 23, 2005 11:50:00 PM

Mark, one thing I saw in the ABC poll and what appears to be in this latest CBS poll are data regarding how closely people are following the story, but without crosstabs of opinion by level of attention payed to the story. Are there differences in opinion by level of attention, and if so, what are they? Isn't that information worth reporting?

Also, note the progression of questions 8, 9, 10, and 13 in the CBS poll. Questions 8-10 do not mention Terri, but Question 13 does. Questions 8 and 9 mention "feeding tube" and not "life support," but notice these phrases: 8: "doctors say brain activity has stopped" and 9: "in a coma with no brain activity."

Questions 8 and 9 clearly do not describe Terri's condition as she is NOT brain dead. She is severely brain damaged and large portions of her brain have atrophied, but the phrases "brain activity has stopped" and "no brain activity" are not accurate in Terri's case. I suggest that this is important because of the progression of the questions.

From what I can tell, none of the preamble explained that PVS does not mean "brain activity has stopped" or that she has "no brain activity." In other words, the questions leading up to the questions specifically about Terri's case seem to be hinting that Terri is brain dead when she clearly is not.

I still don't think these nuances would change public opinion drastically, but it sure seems odd to me that the survey would be written this way. Can you comment about the progression of questions 8-10 and 13 and what you suspect is the purpose of this sequence? (What happened to questions 11 and 12?) Any potential to make less informed respondents think that Terri is brain dead? Could the data displayed by level of attention payed to the story give us any hints? Also, when the results of Q8 and Q9 were reported in the CBS news story, they mentioned "coma" and "vegetative state" but made no mention of the reference to the fact that their questions included the qualifier of "no brain activity." Seems odd to me as well.

A while back Mark I told you that I thought that polling companies should be headed by two highly partisan pollsters to look over each others shoulders and reduce unintentional bias in design and methods. You said that is pretty common. I have to wonder what's going on in this case though. If I were consulting the survey design, I would have immediately raised concerns with using terms like "persistent vegetative state" and a progression of questions that seem to infer that Terri has no brain activity. Not saying that there is some agenda going on here; but, I don't get this one...

Posted by: Rick Brady | Mar 24, 2005 2:17:11 AM

"1. Any summary of "the facts of the case" in a few sentences must be intellectually inadequate, since most cases (including this one) cannot be summarized fairly in a few sentences. Thus any poll question that is prefaced by such a summary is suspect."

This is a circular argument. You say that the summary is inadequate because the issue can't be summarized adequately.

2. The claim that the summary would make Jack Webb proud ("Just the facts, ma'am") is also intellectually inadequate. One can write a summary using undeniably true, or arguably true, statements, and by cherrypicking the facts you use, present a biased view of the case.

The ability to present a "biased view" does not imply that all summaries are biased. More theoretically, any poll result is inherently tied to question wording, question order, response choice, moment in time, etc. so you can then say that asking any question with any set of answers and any order is by definition presenting a "biased view."

The reality is that good pollsters do their best to present neutral surveys that accurately gauge public opinion, and their reputations depend on this ability.

4. The people who use the term "push poll" here are no more (or less) limited to the technical definition of "push poll" than other people are limited to the technical definition of "life support".

From wikipedia.org:

"Life support is a term for a set of therapies to preserve a patient's life when essential bodily systems are not working well enough to be relied upon. Life support therapies utilize some combination of several techniques: enteric feeding, intravenous drips, total parenteral nutrition, mechanical respiration, heart/lung bypass, defibrillation, urinary catheterization and dialysis."

"A push poll is a campaign technique in which a fake poll is used to alter the views of respondents. Push polls are generally viewed as form of negative campaigning. The term is also sometimes used incorrectly to refer to actual polls which test political messages, some of which may be negative. Push polling has been condemned by the American Association of Political Consultants."

A "push poll" is an easily defined term referring to fake polls intended to persuade "respondents." You guys seem to be interested in bias - using "push poll" to refer to a poll which is not completely "objective" (who determines objectivity?) is effectively political rhetoric and is inappropriate for a discussion on who is misleading whom.

"...Phrased this way, I suspect you'd get a much different poll response than the ABC poll."

So in other words you are saying that question wording affects the results of a poll? Thanks for the insight.

Posted by: Chris Kennedy | Mar 24, 2005 4:10:05 AM

Mark,

Good post, which I find much to agree with. I do have some quibbles, which I have blogged here. http://dalythoughts.com/index.php?p=3054

In particular, I do not agree with you that the standard should be "is the wording defensible".

I offer a suggestion for how I think polls conducted for mainstream media outlets should be conducted in my post. The basic idea is to abandon trying to frame questions neutrally, but rather to actively try to frame the question in three ways (one for each side of the issue, and one that splits the difference) and then use split samples.

Regards,
Gerry

Posted by: Gerry | Mar 24, 2005 9:44:32 AM

Mystery Pollster, you seem to entirely miss the bias inherent in questions that tell the person being polled that the husband says Terri would want the tube removed, and the parents think she can still get better. This strongly implies to even a not-casual listener that it is AGREED UPON that Terri's wishes were to be allowed/made to die, and her parents weren't carrying them out because they harbored vain hopes of her getting better. Most people, I would imagine, think that the person's wishes should be carried out, and if Terri had left a living will instructing the removal of food and water, there would be very few objectors other than religious nuts. But in fact, WHETHER OR NOT those were really Terri's wishes is the crux of the matter. The standard of proof as to whether those were her wishes is disturbingly non-existent here. I'm not at all a religious person, and I have a living will which allows me to be taken off life support, but I can tell you that as a former prosecutor, I would never bring a case that depended on the statement of a single witness with a powerful conflict of interest and no corroboration, and if I did, that person would almost certainly be acquitted immediately. (To those who say that the standard of proof here isn't as high as it is in a case where the accused goes to jail for a year -- well, shouldn't it be?) By the way, just to nitpick a little -- most laymen do think of "life support" as being respirators and suchlike. Based on my experiences with my father being on life support at the end of his life, most DOCTORS mean respirators when they speak of life support, and call feeding tubes feeding tubes.

Posted by: Lisa | Mar 24, 2005 11:49:41 AM

Anyone who believed that that Terry should live would not have used that language.

Anyone who believed that her shell should be maintained would not have mentioned that the only functioning part of her brain is the stem, and that Mike the Chicken was kept alive for 18 months despite having his head lopped off right above the stem.

Anyone who believed in the tooth fairy would not want the question on whether the tooth fairy existed to include the fact that parents replace lost teeth with quarters.

There is a reason why "feeding tubes" are considered "life support" under both medicine and the law---and why in the Schiavo case there is no medical or legal distinction between the removal of nutrition through a feeding tube, and the removal of a breathing apparatus.

To suggest that a poll is somehow biased because its question is inconsistent with the campaign of lies and disinformation being raged by the Schindlers and their allies is completely moronic.

In general, informed opinion is far more valuable a measure of public opinion than is opinion held in ignorance of the facts. (Of course, its also helpful to know how stupid people think --- the fact that significant majorities of Bush voters were completely clueless when it came to the facts of WMDs in Iraq and an Iraq-al Qaeda connection is very valuable information, because it does provide hope that this nation's reputation can be redeemed if more of an effort is made to get facts to voters.)

Posted by: p.lukasiak | Mar 24, 2005 1:31:01 PM

Chris, perhaps you should reread my posting. I did NOT use the term "push poll" to describe what ABC did. And I have no idea who the "you guys" are that you lump me with. I'll be happy to defend what I said, but I'm not eager to spend any time defending what you erroneously believe I said, or defend your attacks on the unnamed people you associate me with. Please show a little less sarcasm and a little more intellectual openness.

Posted by: CivilWarGuy | Mar 24, 2005 1:38:02 PM

someone who identifies himself as CivilWarGuy wrote

"And I have no idea who the "you guys" are that you lump me with. I'll be happy to defend what I said, but I'm not eager to spend any time defending what you erroneously believe I said, or defend your attacks on the unnamed people you associate me with."

gee, I guess it was another "CivilWarGuy" who defended those who used the words "push poll" to describe the ABC poll when he wrote....

" 4. The people who use the term "push poll" here are no more (or less) limited to the technical definition of "push poll" than other people are limited to the technical definition of "life support".

Posted by: p.lukasiak | Mar 24, 2005 1:57:39 PM

Mark, I think that your discussion here implicitly endorses a commonly held error about the best way to interpret polling data about matters of public interest. (And this error underlies the criticism of the ABC poll as well.)

The error is the incorrect belief that there is a "right" or "unbiased" way to ask a question about any given public issue. There is no such thing. Everyone who works within the polling field is well aware that small changes in wording can affect the ways in which respondents answer questions. This approach leads us into tortuous discussions of question wording on which reasonable people can differ. Further, as you have pointed out many times in the past, random variation in the construction of the sample or in response rates can skew the results of any single poll away from the true distribution of opinions in the population.

So how do we look at public opinion on an issue such as the Schiavo case? The answer is NOT to find a single poll with the "best" wording and point to its results as the final word on the subject. Instead, we should look at ALL of the polls conducted on the issue by various different polling organizations. Each scientifically fielded poll presents us with useful information. By comparing the different responses to multiple polls -- each with different wording -- we end up with a far more nuanced picture of where public opinion stands on a particular issue. If we can see through such comparisons that stressing different arguments or pieces of information produces shifts in responses, then we have perhaps learned something. Like our own personal opinions, public opinion is not some sort of simple yes/no set of answers; it is complex, and it can see both sides of complicated issues when presented with enough information.

If we were to lock pollsters of all partisan persuasions in a room and force them to pick the "best" question wording on the Schiavo issue, we might end up with everyone asking the same question, but overall we would end up with less information about public opinion, not more. We are better off having the wide variety of different polls, with questions stressing different points of view on the issues, and then comparing them all to one another. This is precisely what you do in your discussion of the ABC poll, but I think you are asking entirely the wrong question -- not "is the ABC wording defensible?" but rather "what does the ABC poll, when compared to other polls with different wording, add to our overall understanding of public opinion on this issue?"

Of course, this sort of contextualizing of polling results is exceedingly rare in the media. Much more common is the front page story saying "here is our poll, and here is what it found, and it is a true representation of public opinion" -- and by implication, no other poll matters. Intellectual honesty is trumped by competition. The best we usually get are vague generalizations of all of the polls lumped together ("polls have consistently shown disapproval of Congress' actions"), and even those generalizations almost never appear in the initial story trumpeting the "exclusive" poll fielded by the newspaper/network itself.

The end result is that even those who pay close attention to the news media and the chattering classes often have very little real understanding of how to interpret polls in a thoughtful way -- which is one of the reasons your blog is so valuable.

P.S. Polls which attempt to predict election results are a rather different kettle of fish, for two important reasons: (1) Pollsters have been experimenting with questions wording for over 50 years and can keep wording the same regardless of the issues in a race; and (2) There is an actual real-world "check" on pollsters' work in the form of the actual election results. Neither of these characteristics apply to polling about issues of public interest.

Posted by: Professor M | Mar 24, 2005 1:59:56 PM

With respect, I think that the term 'life support' is biased, but that it doesn't have much impact on the results of the survey.

Of course...I think it is more biased that the questions tend to be phrased as 'her husband wants this' versus 'her parents disagree.' It makes it appear that there is a question about the facts, and it is just 'he said/she said'

Posted by: Rich Gibson | Mar 24, 2005 2:06:55 PM

On Lisa's comment:

The Schiavo case is NOT one that relies on a single witness, whose testimony is uncorroborated. That is the most important fact that I think is consistently left out of treatments of this issue. There were FIVE people who reported hearing Terri comment on being kept alive, including her mother. Two of those people heard her comment on the case of Kathleen Ann Quinlan, and while they witnesses failed to agree on the timeline, the notoriety of the case makes plain that she would have said these things as a girl of 11 or 12. Furthermore, her comments were related to what others should do for Quinlan, not what her own views were.

HOWEVER: there are three other people (Michael, his brother and his sister) who gave corroborating testimony of multiple conversations in which Terri specifically told them--in the context of funerals for people who had died after being kept on life support, mind you--that she did not want to die in such a fashion.

Judge Greer notes in his ruling that a single statement by the husband might not have sufficed as evidence to grant her relief--but supporting testimony did indeed bring the level of proof across the "clear and convincing" threshhold.

Posted by: torridjoe | Mar 24, 2005 2:14:42 PM

uh, the penultimate paragraph should end:

"...that she did not want to LIVE in such a fashion."

Posted by: torridjoe | Mar 24, 2005 2:19:34 PM

uh, the penultimate paragraph should end:

"...that she did not want to LIVE in such a fashion."

Posted by: torridjoe | Mar 24, 2005 2:19:54 PM

You see, the problem is that the facts regarding the discrepencies between the two sides has almost never been told in the regular, main stream media, and god forbid you would do so here.

My problem with the ABC poll os not so much that they used the term life support, it's that they say "Doctors say she has no consciousness..." in their second question. The greater percentage of the public at large is not well enough acquainted with all the facts of this case, and it's an important fact that many doctors DO NOT AGREE with her diagnosis, or at least say such a diagnosis can not be accurately reached with the tests that have been done, which in essence means Terri could very well have a level of consciousness.

Did the poll tell people that? No, it states as fact that doctors say she has no consciousness. It leaves no room for the possibility that she may, and that many respected doctors feel this way.

Tell me, how is that fair? How is that NOT misleading? What's that? You're saying that the CAT scna WAS sufficient to diagnose her? Well, then why are some doctors saying that simply not true?

Once you start the poll like that, you could expect that people would say pull the tube. Why not, if she has no consciousness? The problem is that the actual circumstances of the case were not presented AT ALL, only Michael's "facts" were given, and the dissenting view was not even said to exist.

That is just plain misleading trickery.

Okay, now lets here all your silly reasons why the poll is still okay...

Posted by: jeff | Mar 24, 2005 2:36:03 PM

Poll results have little value without corresponding quantitative values for sampling error.
Sampling error values (margin of error) however, are based on random samples.
Since truly random samples are not possible under any circumstances (particularly with telephone calls where the day of the week, the time of day, the demographics of phone owners, the refusal of certain people to answer the questions, the fact that some people simply lie, etc.), the results of all polls are biased, ie.not completely random.
So called "margins of error" are therefore nothing but a myth used by pollsters to deceive the public into accrediting polling results with a unwarranted degree of precision.

Posted by: Downeaster | Mar 24, 2005 3:50:10 PM

I fear I must argue with Mr. Kaus on your site. It does not seem that he is aware of what's involved in "removing a respirator". He posits a death within minutes, apparently believing that a machine is simply taken out.

That's not the way it works. My siblings and I attended our father as his respirator was turned down from whatever level of assistance it was providing. This process took several hours, with the level being diminished every 30 minutes or so. Eventually, his heart gave out before the dial was reduced to zero. He was in no extended discomfort, but those hours were agonizing for his children.

Rhetorical question: should polling questions be devised for two separate groups? One for people who don't know the history of the item being polled, and one for those who do?

I'd also love to see a state-by-state breakdown on the numbers. According to my local legislators' offices, even though it's a Republican district, they were being flooded by calls from voters saying "back off the governor's legislation: it's wrong."

Posted by: John Burgess | Mar 24, 2005 5:56:49 PM

"Rhetorical question: should polling questions be devised for two separate groups?"

Um, no, but they should either contain all of the facts or none of the facts...not half the facts that only show one of two opposing viewpoints. This is even more true in an instance such as this, where the omitted facts and distorted tone can so greatly influence a person's take on the situation...and therefore their poll answer.

Posted by: jeff | Mar 24, 2005 6:28:38 PM

Sorry to back up so far, but to Chris' post:

"From wikipedia.org:

"Life support is a term for a set of therapies to preserve a patient's life when essential bodily systems are not working well enough to be relied upon."

Last I heard, Terri Schiavo's gastrointestinal tract works just fine when it has food introduced to it (as does mine most of the time). Difficulty swallowing is not a problem with an essential bodily system (note that I said 'difficulty swallowing', as she can swallow and be fed thin solutions from what is essentially a baby bottle, but she risks choking). Also, notice that it refers to a "set of therapies" meaning that any singular therapy mentioned may/may not be considered "life support". This definition from wikipedia would seem to damage your own arguement, thanks.

Posted by: Doug | Mar 25, 2005 4:58:08 AM

I am so sick of hearing about the Schiavo case. This is a personal decision by Terri and her husband and no-one else, especially not the President/Governor or Legislature.

I hope she dies quickly and, as the polls show, this never becomes a media/social conservative/religious issue again. It is an abuse of power and the media.

Thanks

Posted by: Stan | Mar 25, 2005 12:49:52 PM

Personally, I think that all the polling is appalling. There is a family about to lose their daughter because she is being deprived of nourishment. If you have ever experienced a similar situation with a loved one what you think you would do and what you do may be different. I don't know what kind of a state Ms Schiavo is in, but I guarantee you that neither do the neuralogist!!! If you spent two months in a critical care unit watching neuralogists walk in a patient's room, in a large group, and start screaming at the patient, pinching them as hard as they can, you would soon realize that the patient knows when they walk in and will intentially "space out", if not go into a hyperactive state. Seconds later, the same patient will be quietly and calmly responding to family members. The word in the hospital is that neuralogist are the "doctors of doom and gloom". The only test that proves anything is if the person is "brain dead", that is the brain stem is not functioning. That is not the case here. The fact is, the neuralogist are making a measured supposition by playing the odds. They may as well be tossing the dice. All they will tell you is that some percentage will recover at some percentage, but there is little hope. What we should be talking about is the rights of the disabled person to receive not just supportive care, but physical and occupational therapy to see if they can improve. A hospice does not provide that. Terri has been left to disentegrate with nothing more than supportive care. Just look at her contracted hands. That is the first indication that she has been neglected. With proper care and therapy this does not have to happen. Her husband made the decision to let her stay in this condition and as a result she has not been allowed the kind of therapy that would have proved whether or not she can improve. What is wrong is a law that allows a husband who has essentially abandoned his wife for another woman with whom he now has children, to be her guardian. The very fact that he chose this path is why her parents should be her guardian. It is amazing that "man" is still considered by the law to be "head of the household". He is her husband in name only and he should walk away and leave her care in the hands of those her love her.

Posted by: Sheila | Mar 25, 2005 3:35:29 PM

The comments to this entry are closed.