Watch Out for the Easy Out

Legacy blog posts Polls in the News

As of last week (before yesterday’s indictment news) what portion of Americans believed “Bush administration officials” did something “illegal or unethical” in connection with the leaking of a CIA operative’s identity?  CNN’s summary of the latest CNN/USAToday/Gallup poll may have left some readers a bit confused.  A quick lesson in survey methodology may help clear up the confusion.

The CNN story leads with the following sentence:

Only one in 10 Americans said they believe Bush administration officials did nothing illegal or unethical in connection with the leaking of a CIA operative’s identity.

But then three paragraphs later, the story tells us:

The poll was split nearly evenly on what respondents thought of Bush officials’ ethical standards — 51 percent saying they were excellent or good and 48 percent saying they were not good or poor.

Huh?  Fifty-one percent (51%) of Americans rate the ethical standards of Bush administration figures as “good” or “excellent,” yet only 10% say they did nothing “illegal or unethical” in the Plame case.  How can that be?

One possible if unlikely explanation is that the more general question came first.  Perhaps, even before the announcement of the Libby indictment, Americans were evenly divided on the general ethical standards of Bush administration officials even though only 10% believe they did nothing “illegal or unethical” in the specific example of the Plame case. 

A better and more likely explanation has to do with the way Gallup asked the question that produced the 10% result and the way CNN chose to present it.  Let’s take a look at the question:

As you may know, several members of the Bush administration have been accused of leaking to reporters the identity of a woman working for the CIA.  Which of the following statements best describes your view of top Bush administration officials in these matters – some Bush administration officials did something illegal, no Bush administration officials did anything illegal, but some officials did something unethical, or no Bush administration officials did anything seriously wrong?

39% did something illegal

39% unethical, not illegal

10% nothing wrong

12% had no opinion

This question forces a choice between three alternatives with one category (not illegal but unethical) clearly in the middle.  Survey methodologists have long debated the merits of presenting an expclit middle alternative choice in survey questions.  On the one hand, the true attitudes of the respondents may sometimes fall in the middle.  On the other, some respondents may use a middle category as an “easy out” rather than thinking hard about the issue at hand.  The academic research on this issue is mixed (see O’Muircheatough, Krosnick and Helic, 2000, for a summary), but even then the academic research MP knows is not exactly on point with this example, as it tends to focus on very concise answer categories, such as whether the government should spend “more, about the same or less” on various programs.

What the academic literature tells us more clearly is that “easy out” behavior by respondents (something methodologists call “survey satisificing”) appears to occur more often when questions are wordy and confusing.  Ironically, a recent working paper based on experiments by the Gallup Organization (Holbrook, Krosnick, Moore and Tourangeau, 2005) on “order effects” (whether the order of answer categories affects the results) reached the following conclusion (pp. 22-23):

We found that response order effects were more common among questions that were more difficult to comprehend, when the response options were complete sentences instead of single words or phrases, and among questions involving response options that were not mutually exclusive. These findings are consistent with a large body of evidence from previous research that have found response order effects to be stronger for longer questions, questions with longer response options, and those with more difficult language (Bishop and Smith 2001; Payne 1949/1950; although McClendon 1986a found no such relation).

The authors of the report concluded with this advice for pollsters (p. 27) :

Our findings have a number of implications for survey research practice, though they are hardly surprising.  Because response order effects are stronger in questions that are more difficult to comprehend, researchers should strive to keep their language as simple as possible.  Advice of this sort is offered in numerous research methods textbooks, but a cursory examination of many survey questionnaires suggests that there may still be room for researchers to heed this advice more faithfully.  It is not uncommon to see questions laden with social science jargon and long sentences involving many multi-syllabic words.  Because survey researchers are usually well-educated, they may find such questions to be easily understandable. 

With all of that in mind, let’s consider that Gallup question again.  Is it likely that a large number of Americans considered the actions of “Bush administration officials” in the Plame affair unethical but not illegal as of last week?  Certainly.  So posing a question with a middle category in this case seems reasonable.  However, the question they wrote is a it long and wordy side.  For example, it repeats the phrase “Bush administration officials” three times in one sentence, throwing in “officials” a forth time.  Why not simply ask whether Bush administration officials “did something illegal, did something legal but unethical or did nothing seriously wrong?”   Another complication is that the middle category poses something of a double-negative: not illegal but unethical. 

MP wonders how many respondents found this question confusing or hard to follow.  MP also wonders whether some heard the question as having four choices rather than three.  If one does not listen carefully the middle option may sound like two distinct choices:  “no Bush administration officials did anything illegal, but some officials did something unethical.”  Is it possible that some respondents replied that they preferred the second option because they thought it said only that Bush officials “didn’t do anything illegal?” 

MP encourages readers to try this experiment:  Read the question aloud to someone and ask them to answer it.  Better yet if that person lacks a college degree.  Are they confused?  Are they confused?  Do they ask you to read the question again?   

In a perfect world of unlimited budgets and no time constraints, survey organizations would conduct extensive pre-testing to check against such confusion.  MP has no knowledge of how carefully Gallup pre-tests its surveys, but his experience is that most pollsters lack the time and budget to pre-test as thoroughly as they would like. 

So what can we make of results of this question?  Here is what we know:

  • On another simpler question on the same survey, 51% rated the “ethical standards of top administration officials” as excellent or good, 48% as not good or poor.   
  • On the same question more than half the respondents chose one of the two middle categories: 11% rated the ethical standards excellent, 40% good, 19% not good and 29% poor. 
  • When Gallup tabulated responses to the longer question by party identification, the “did something illegal” response follows the pattern we would expect:  much higher among Democrats (55%) than Republicans (19%) with independents somewhere in the middle (45%). 
  • However, preference for the middle choice – unethical not illegal – was similar among Democrats (36%), independents (36%) and Republicans (44%).

MP’s admittedly subjective conclusion:  The middle category of the three way illegal/ unethical question may probably included many respondents without well formed opinions in any direction.  The 39% who believe Bush officials did something illegal is the most reliable and newsworthy result from this question, while the lines between other categories seem more blurry.  The USA Today story and the Gallup release both put appropriately greater emphasis on the 39% result.  CNN would have done better to follow their lead, rather than hyping the “only 10%” result.

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.