The RT Strategies Iraq Poll

Yesterday’s front-page analysis of President Bush’s Iraq speech earlier this week by the Washington Post‘s Peter Baker included a reference to some recent survey data: 

Administration officials believe much of the public is still eager for victory and open to persuasion if the president can make the case that he has made progress. They took heart in a survey last week by RT Strategies, a bipartisan polling firm, that found that 49 percent of Americans favor bringing troops home when only "specific goals and objectives" are met, 30 percent want a fixed timetable for pulling out and 16 percent support immediate withdrawal. The middle 30 percent, they figure, is the real political battleground.

In an item in last Sunday’s "Politics" column, the Post‘s Chris Cillizza called the poll "one of the few pieces of supportive news the administration has had lately on Iraq."  He led with the finding of another question showing that 70% of adults "said that criticism of the war by Democratic senators hurts troop morale." Cillizza also looked at the poll in a blog item (that included links to results, crosstabs and an RT Strategies press release).   

Not surprisingly, these findings have already been the subject of much debate on Cillizza’s blog and elsewhere, partly because of the perception that they contradict the findings of other polls.  I do not believe they do.  Instead they help underscore that when it comes to prospective policy on Iraq, polls are inconsistent consensus is elusive.  This post (which continues after the jump due to its length) reviews the nuts and bolts of the RT Strategies Poll and how its results fit into the bigger picture regarding public opinion on the Iraq War.

What is the RT Strategies "Omnibus" Poll?

First, let’s review what we know about this poll and how it was conducted.  Last week, I linked to a new national survey, a "national omnibus poll" conducted by a new bi-partisan research company named RT Strategies.  The company is a partnership of Democrat Thomas Riehle and Republican Lance Tarrance.  Riehle worked most recently as head of IPSOS Public Affairs, where he helped create the AP/IPSOS survey.  Earlier in his career he worked variously for Democratic pollsters Peter Hart and Pat Caddell.  Tarrance founded and ran the well-known Republican firm that bears his name (originally Tarrance and Associates, now the Tarrance Group) although he is no longer affiliated with that company. 

[Interest declared:  Riehle is a personal friend and an occasional commenter on this site]. 

Their first survey — the one that is getting all the attention — is part of something called an "Omnibus Poll."   Omnibus in this context means that RT Strategies conducts one poll, but allows multiple paying clients to buy questions.  Here is the way the National Journal’s Gregg Sangillo described it ("Consulting Game," 11/19/05): 

The twist is that in any given week, any outside groups, such as political party organizations or lobbying firms, can ask RT Strategies to pose additional questions as part of the poll. Riehle says that traditionally, if an organization wanted a poll question answered, it would have to "call up a company and do a whole poll. Well, now you’re talking real money. We’re saying, we have a vehicle, it’s going to be going out into the field every week, [so] just ask the questions you need to ask."

MP readers may recall that their first survey also included questions written and analyzed in partnership with Charlie Cook and his Cook Political Report.  However, despite some initial confusion to the contrary, Cook had nothing to do with the questions on Iraq.  They were not part of the Cook section of the poll.  "I did not even know they even existed," Cook told me via email, "until after the survey was completed."  [For those interested in Cooks questions, his analysis, the top-line results, crosstabs and a press release from RT Strategies are posted on Cook’s website].   

Such are the pitfalls of omnibus surveys.  A client can buy a question or two without having to bear the costs of fielding a complete survey.  The disadvantage is that others questions may precede it that a client knows nothing about (except for those with the clout to insist that their questions come first). 

Omnibus polls are not new.  The AP/IPSOS poll (which Riehle helped create) is part of a regular omnibus poll in which both AP and other paying clients ask questions.  According to the folks at AP, their questions always come first.   The "Battleground Poll" conducted by the bipartisan team of Republican Ed Goeas (of Lance Tarrance’s old company, the Tarrance Group) and Democrat Celinda Lake (of Lake, Snell, Perry, Mermin) is also an omnibus survey that includes questions fielded for private clients of the respective firms. 

Generally, commercial omnibus surveys include questions on a wide variety of different topics, which generally cover consumer products and services rather than political issues.  However, it is always possible that a question asked early in an omnibus survey might have some biasing effect on a question asked later.  Omnibus poll clients put their trust in the pollster to avoid ordering questions in ways that would create an obvious bias. 

To be fair to the omnibus pollsters, I should point out that surveys conducted by the major national news media outlets also typically ask about a wide variety of different issues in the course of a single survey.  They too must make subjective judgment calls about how to order the questions and whether that order may unintentionally influence the results.  While academic survey methodologists have studied "order effects" using split sample experiments, their findings provide no more than a rough guide to survey practitioners.  Public and commercial pollsters rarely have the time or budget for experimental research.  Thus, we rely mostly on experience and judgment in determining how to order questions and, in tracking surveys, whether inserting new questions or cutting old ones will create unexpected "order effects."

Who Was the Client for the Iraq Questions?

That was the question I wondered about when I first saw the results. According to Riehle, there was no client.  "Lance and I wrote them," he told me when I called, adding that they wanted to include some newsworthy questions to help generate buzz for their new survey.  In a comment left on Cillizza’s blog, Riehle also added the following: 

Lance and I intended to touch on the most emotional topics in the news in our choice of questions to ask (see, for example, the intense and often emotional debate in the House these past few days!), and we hope the questions themselves are pretty straightforward in how they measure the division of public opinion on those particular highly-charged questions.

Which brings us to the buried yet nonetheless critical question…

What Should We Make of RT Strategy’s Iraq Questions?

In his Sunday item, the Post’s Cillizza implied that these results contradict other polls that show "significant majorities believe it was a mistake to go to war, as well as rising sentiment that Bush misled Americans in making the case for it."  Is that a reasonable supposition?

To consider the questions we need to step back for a moment and think about public opinion more generally.   Most Americans know that George W. Bush is our president, that we are at war in Iraq, and that most elected officials are either Democrats or Republicans.  Thus, on these topics, most Americans possess general opinions.  So when pollsters ask questions about how well the President is doing his job, whether the war in Iraq was a good or bad idea, or the general strengths and weaknesses of the political parties, pollsters can tap real pre-existing opinions.  Questions on these general topics tend to be consistent across polls, even with slight variations in question wording. 

However, most Americans have far more limited knowledge of more specific aspects of public policy and the day-to-day debate that captivates policy makers, political junkies and most MP readers.  In a poll released Wednesday night, for example, Gallup found that only 10% of Americans reported seeing Bush’s speech on Wednesday while 66% had heard nothing about it (subscription-only Gallup release here). 

When pollsters move beyond general ratings to more specific questions about policy – as we do in almost every public political poll – we move to shakier ground.  Here Americans often lack preexisting attitudes, yet most will work to answer our questions, often forming opinions on the spot based on the text of the question.  When that happens, responses can be very erratic and contradictory across polls.  Very small variations in wording, the number of answer choices offered or the order of the questions can result in big and often surprising differences in the results. 

With that in mind, consider the three RT Strategies questions: 

Thinking about the war in Iraq, when Democratic Senators criticize the President’s policy on the war in Iraq, do you believe it HELPS the morale of our troops in Iraq or HURTS the morale of our troops in Iraq? (IF HELPS/HURTS, ASK:) And do you believe it (HURTS/HELPS) morale A LOT or just SOME

44% hurts a lot
26% hurts some
6% helps some
7% helps a lot
17% not sure

When Democrats criticize the President’s policy in Iraq, do you believe they are (ROTATE) Criticizing the President’s policy because they believe their criticisms will help the United States’ efforts in Iraq, OR, Criticizing the President’s policy to gain a partisan political advantage?

31% believe will help
51% to gain advantage
6% some of both (volunteered)
6% neither (volunteered)
7% not sure

And thinking about the future of our policies in Iraq, do you believe the U.S. military should…. (ROTATE FIRST TWO, ALWAYS ASK "Set a fixed timetable" last) Withdraw our troops immediately, regardless of the impact OR Withdraw our troops as the Iraqi government and military meet specific goals and objectives OR Set a fixed publicly available timetable for withdrawal.

16% withdraw immediately

49% withdraw when goals met

30% set fixed timetable
3% none (volunteered)
2% not sure

A few reactions:  First, all three of these questions fall into that second category of issues about which many Americans appear to lack preexisting attitudes.  Non-attitudes are most evident on the morale question (something that Armando at DailyKos picked up on).   The telltale clue is that 17% were completely unable to answer the question, a sure sign that many more came up with an answer on the spot.   The fact that nearly a third chose one of the softer "some" categories (26% hurts morale "some," 6% helps "some") is consistent with that argument.  Also consider the respondent who believes such criticism neither hurts nor helps troop morale, but does not realize that "neither" is an o[tion.  Odds are good they will end up in the "hurts a little" category. 

On the partisan advantage question, nearly one in five respondents (19%) could not choose between the two offered answer categories.  Finally, for reasons that I’ll discuss below, I’d argue that the large number of respondents in the middle category of the future policy question (49%) suggests that it was an attractive choice for those respondents who were simply not sure how to answer.   

Now MP is not averse to survey questions that offer new information and push respondents a bit to see where they might stand in debates they have not followed closely.  And in this case, the results of the RT "morale" and "criticism" questions are more or less consistent with the similar questions asked elsewhere.   For example, a Fox News poll in early November found that 58% of Americans agree that those "who describe U.S. military action in Iraq as a mistake" are "hurting U.S. troops."  Only 16% believed they were "helping."  The rest had mixed opinions (9%), believed the criticism had no effect (9%) or could not answer the question (8%). 

It is also worth noting that Americans tend to dismiss much of the debate in Washington as attempts to gain "partisan advantage," so the results of the RT question are not particularly surprising.  For example, back in September (9/8-11), Gallup asked about politics in the context of Hurricane Katrina:

"Do you think Democrats who criticize the way the Bush Administration has handled the hurricane response mainly want to find out what went wrong, or mainly want to use the issue for political advantage?"
36 find out, 60% use for advantage, 4% unsure

Seven years ago, ABC News and the Washington Post asked a similar question about the impeachment of President Clinton with nearly identical overall results:

"Do you think the House voted to impeach Clinton on the basis of the facts of the case, or on the basis of partisan politics?"
36% facts of the case, 61% Partisan politics, 3% no opinion

Questions that push respondents to consider questions for which they do not have pre-existing opinions do have a role in opinion research (one that should not be labeled as a fraudulent "push poll" — but that’s another subject for another day).  However, in those instances pollsters need to take care to provide respondents with new information in a way that does not bias subsequent questions.  For that reason, I am a bit surprised that RT Strategies asked two questions that mirrored the Bush administration talking points just before asking respondents their preference about prospective Iraq policy.  Would the responses to the third question have been different if they followed a question about say, whether Bush "intentionally misled the American people about the presence of weapons?"  We will never know, but it certainly seems possible that they would. 

To be fair, Gallup asked a very similar question a few weeks ago (11/11-13) with similar results:

"Here are four different plans the U.S. could follow in dealing with the war in Iraq. Which ONE do you prefer? Withdraw all troops from Iraq immediately. Withdraw all troops by November 2006 — that is, in 12 months’ time. Withdraw troops, but take as many years to do this as are needed to turn control over to the Iraqis. OR, Send more troops to Iraq." 
19% withdraw now, 33% withdraw within 12 months, 38% take as long as needed, 7% send more troops, 3% unsure. 

Note that Gallup showed 19% ready to withdraw immediately; RT Strategies show 16%.  Gallup shows 52% supporting withdrawal either immediately or within 12 months, RT shows 46% support withdrawal either immediately or on a fixed timetable. 

Having said this, I want to caution readers against taking these these prospective policy questions  at face value.  I also tend to agree with those who argue that the questions on the RT poll are, in essense, the wrong questions, that other measures give a better sense of true, pre-existing opinions on the Iraq War.  This is not necessarily a criticism of Riehle and Tarrance, merely a caution that focusing on these three questions alone can give a misleading impression.  For example, review the questions asked since Labor Day as posted by the Polling Report and you will find some highly consistent results: 

• Approval of Bush’s handling of the war in Iraq varied between 32% and 36%, with disapproval between 62% and 65%, as measured by six different pollsters.

• Differently worded questions about the worthiness of the war (asked by Gallup, CBS, ABC/Washington Post and NBC/Wall Street Journal) found between 31% and 40% that found the war worth the cost and between 52% and 60% that said it was not. 

• Differently worded questions about whether the decision to go to war was right or wrong (asked by Gallup, CBS and the Pew Research Center), found 42% to 45% who say the US made the right decision in going to war, between 50% and 54% who say we made the wrong decision or should have stayed out. 

However, look at the range of questions about prospective policy and the results are all over map.  Here is a sampling (full details at the Polling Report):

CNN/USA Today/Gallup  (11/30):  "If you had to choose, which do you think is the better approach for deciding when the U.S. should withdraw its troops from Iraq: to withdraw U.S. troops only when certain goals are met, or to withdraw U.S. troops by a specific date and stick to that time-table, regardless of conditions in Iraq at that time?"
59% when goals are met, 35% by a specific date, 6% unsure

FOX News/Opinion Dynamics (11/ 29-30):  "Do you think there should be a publicly announced timetable for withdrawing U.S. troops from Iraq?" 47% should, 41% should not, 12% unsure

Harris (11/8-13):  "Do you favor keeping a large number of U.S. troops in Iraq until there is a stable government there OR bringing most of our troops home in the next year?"
35% wait for stable government, 63% bring home next year, 3% unsure

FOX News/Opinion Dynamics (11/8-9): "What do you want U.S. troops in Iraq to do? Do you want them to leave Iraq and come home now or do you want them to stay in Iraq and finish the job?"
36% leave now, 55% finish the job, 9% unsure

NBC News/Wall Street Journal (11/4-7):  "Do you think that the United States should maintain its current troop level in Iraq to help secure peace and stability, or should the United States reduce its number of troops now that Iraq has adopted a constitution?"
36% maintain level, 57% reduce number, 4% both depends, 4% unsure

ABC News/Washington Post (10/30-11/2):  "Do you think the United States should keep its military forces in Iraq until civil order is restored there, even if that means continued U.S. military casualties; or do you think the United States should withdraw its military forces from Iraq in order to avoid further U.S. military casualties, even if that means civil order is not restored there?"
52% keep forces in, 44% withdraw forces, 4% unsure

CBS News (10/30-11-1): "Should the United States troops stay in Iraq as long as it takes to make sure Iraq is a stable democracy, even if it takes a long time, or should U.S. troops leave Iraq as soon as possible, even if Iraq is not completely stable?" 43% stay as long as it takes, 50% leave ASAP, 7% unsure

Pew Research Center (10/6-10): "Do you think the U.S. should keep military troops in Iraq until the situation has stabilized, or do you think the U.S. should bring its troops home as soon as possible?" 47% keep troops, 48% bring home ASAP, 5% unsure

So there we have it:  A consistent majority of at least 60% of Americans now disapproves of President Bush’s performance on the Iraq war and believes it not worth the cost.  A smaller majority now says that the war was a mistake.  The consistency of the results suggests these are real attitudes, not opinions formed on the spot in the response to the language of the question. 

However, when pollsters ask what we should do next in Iraq, results are highly inconsistent.  Support for leaving sooner varies anywhere from 35% to 63% on the questions listed above.  Support for staying the course (in one form or another) varies from 36% to 59%.  Ask a question with three or more options (as RT Strategies and Gallup did above) and, not surprisingly, at least a third of Americans opts for the middle category.  When it comes to prospective policy, Americans – like their leaders – are divided and collectively not quite sure what to do next.