ABC/Washington Post on Judicial Nominees

Legacy blog posts Polls in the News Weighting by Party

The conservative wing of the blogosphere took great exception yesterday to the latest survey from the Washington Post and ABC News that gave front page play to the assertion that “a strong majority of Americans oppose changing the rules to make it easier for Republican leaders to win confirmation of President Bush’s court nominees.”   The complaints fell into two categories, (1) that the sample was unrepresentative and (2) that the questions “changing the rules” was biased.  MPs quick take is that the former complaints are largely unfounded, the latter debatable.   Let’s take a closer look.

1) Biased sample?  Our friend Gerry Dalyes (of Dalythoughts) nicely summarized the first grievance [though as he points out in the comments section, he did not endorse it]: ]

The Ankle Biting Pundits , Erick at Red State and Powerline have all noted (hat tip to Michelle Malkin that while the 2004 exit polls showed that the parties were at parity among voters, the sample in this poll is not; it includes 35% Democrats and 28% Republicans- a 7 point advantage for Democrats.

The problem with this complaint is that ABC News and the Washington Post — like most polling organizations — surveys all American adults, not just registered or likely voters.  The voting population is slightly more Republican than the population of all adults.  Screening for voters is appropriate in a pre-election survey intended to track the campaign or forecast the outcome, but a survey of what “Americans” think ought to survey, well,  all Americans.   Even if you disagree, the issue is not one of “bias” or “over-representation” but of a difference in the population surveyed. 

Among all adults, as opposed to registered or likely voters, most survey organizations have shown a slight Democratic advantage in party ID over the last year, consistent with the ABC/Post results.  I put together the following table that averaged data from 2004 and year-to-date 2005 when available:  The surveys from CBS/New York Times, Harris, the Pew Research Center and Time/SRBI all show a Democratic advantage of two to six points.  Gallup (subscription required) is the exception, showing party ID at parity. 

More to the point is this sentence in the ABC analysis:

Thirty-five percent of respondents in this survey identify themselves as Democrats, 28 percent as Republicans, about the same as the 2004 and 2005 averages in ABC/Post polls. It was even on average, 31 percent-31 percent, in 2003 [emphasis added].

If anyone from ABC or the Post is reading, it would be helpful to see those averages from 2004 and 2005.  Nonetheless, considering the ABC/Post poll’s +/-3% sampling error, the party ID results are within range of the results for the other surveys from 2005 presented above (with the exception of Gallup’s 35% GOP number), though they do look a point or two more Democratic than the average of the other surveys. 

If we were confident that this small difference resulted from random chance or some sort of sample bias aloine, we would want the ABC/Post pollsters to weight their data to correct it.  The problem is that the difference could be the result of a slight variations in question wording, in the content of earlier questions that might affect responses the party ID question the end of the survey, or perhaps they reflect a small real but momentary change in party identification.  If the difference is just about sampling error or sample bias, weighting could make the survey more representative.  If the difference is about any of the other issues, weighting would make it worse. 

The irony of all this — one likely not lost on other pollsters — is that the Washington Post enabled this criticism by breaking with past practice and putting out a PDF summary that included complete results not only for party identification and ideology, but also for the full list of demographics.  MP commends Richard Morin and the Washington Post for taking this step, even though it seems to be bringing them only grief. 

Yes, consumers of poll deserve this level of transparency.  Yes, it is appropriate to ask tough questions about how well any poll represents the nation.  But leaping to the conclusion that the sample composition is “ridiculously bad” (Ankle Biting Pundits) or that it shows “egregious” bias (Powerline) is just flat wrong. 

2) Biased question? – The second category of complaint took issue with the wording and context of the question that was the focus of the coverage:  “Would you support or oppose changing Senate rules to make it easier for the Republicans to confirm President Bush’s judicial nominees?” 

Judicial filibuster is an example of the type of issue that makes pollsters lives miserable. The underlying issue is both complex and remote.  Few Americans are well informed about the procedures and rules of the Senate, and few have been following the issue closely (only 31% tell robo-pollster Scott Rasmussen they are following stories on the judicial nominees “very closely”).  So true “public opinion” with respect to judicial filibusters is largely unformed.   When we present questions about judicial nominees in the context of a survey interview, many respondents will form an opinion on the spot.  Results will thus be very sensitive to question wording.  No single question will capture the whole story, yet every question inevitably “frames” the issue to some degree. 

To MP, the most frustrating bias in media coverage of polling — be it mainstream or blog — is the pressure to find a settle on a single question as the ultimate measure of “public opinion” on any issue.  In a sense, public opinion about issues like the judicial filibuster is inherently hypothetical.  Many Americans, perhaps most, lack a pre-existing opinion.  If we want to know how Americans will react to some future development (or whether they will react at all), no single question can tell us what we need to know. 

The best approach in situations like these is to follow the advice of our old friend, Professor M:

The answer is NOT to find a single poll with the “best” wording and point to its results as the final word on the subject. Instead, we should look at ALL of the polls conducted on the issue by various different polling organizations. Each scientifically fielded poll presents us with useful information. By comparing the different responses to multiple polls — each with different wording — we end up with a far more nuanced picture of where public opinion stands on a particular issue. If we can see through such comparisons that stressing different arguments or pieces of information produces shifts in responses, then we have perhaps learned something

So what can we learn from different polls on this issue?  The PollingReport has a one page summary that includes most recent polling on the issue (including survey dates and sample sizes): 

In mid-March, Newsweek found 32% approved and 57% disapproved changing the rules regarding filibusters with the following question:

U.S. Senate rules allow 41 senators to mount a filibuster — refusing to end debate and agree to vote — to block judicial nominees. In the past, this tactic has been used by both Democrats and Republicans to prevent certain judicial nominees from being confirmed. Senate Republican leaders — whose party is now in the majority — want to take away this tactic by changing the rules to require only 51 votes, instead of 60, to break a filibuster. Would you approve or disapprove of changing Senate rules to take away the filibuster and allow all of George W. Bush’s judicial nominees to get voted on by the Senate?

At the beginning of April, the NBC News/Wall Street Journal poll found 40% who wanted to eliminate the filibuster and 50% wanted to maintain it, when they asked this question:

As you may know, the president of the United States is a Republican and Republicans are the majority party in both houses of Congress. Do you think that the Republicans have acted responsibly or do you think that they have NOT acted responsibly when it comes to handling their position and allowing full and fair debate with the Democrats?

Then there is this week’s ABC/Washington Post survey that found 26% supporting a rule change to make it easier for Bush to win confirmation of his judicial appointees and 66% opposed.  The ABC question had two parts:

“The Senate has confirmed 35 federal appeals court judges nominated by Bush, while Senate Democrats have blocked 10 others. Do you think the Senate Democrats are right or wrong to block these nominations?”  48% said “right,” 36% said wrong, 3% both and 13% were unsure

“Would you support or oppose changing Senate rules to make it easier for the Republicans to confirm Bush’s judicial nominees?” – 26% support, 66% oppose, 8% unsure.

In an online survey, Rasmussen Reports asked a national sample several different questions.  Unfortunately, they did not release the verbatim language.  The following comes from language in their online release: 

“Forty-five percent (45%) of Americans believe that every Presidential nominee should receive an up or down vote on the floor of the Senate. That’s down from 50% a month ago.”

“When asked if Senate rules should be changed to give every nominee a vote, 56% say yes and 26% say no. A month ago, those numbers were 59% and 22% respectively”

The Republican polling firm Strategic Vision asked the following questions this past week of sample of registered voters in Florida:

Do you approve or disapprove of a Republican plan in the United States Senate to limit Democratic filibustering of judicial nominations and allow a vote on the nominations?  Florida registered voters: Approve 44%, Disapprove 33%, Undecided 23%.

Do you approve or disapprove of Democratic filibusters of President Bush’s judicial nominations in the United States Senate? Florida registered voters: Approve 28%, Disapprove 57%, Undecided 15%

One thing largely missing in the questions asked by public pollsters is a better sense of how informed and engaged Americans are in this issue.   So far, only Rasmussen has asked, “how closely have you been following the issue?”  Unless I’ve missed it, no one has asked for a rating of the importance of the issue as compared to issues like health care, Social Security, Terrorism, Iraq, etc. 

In the same vein, MP wishes that a public poll would ask Americans an open-ended question about this issue.  It would first ask, “have you heard anything about a controversy involving President Bush’s judicial nominations?”  Those who answer yes would then get an open-ended follow-up: “What specifically have you heard?”  The answers would help show how many have pre-existing opinions that demonstrate worry about conservative nominees or about the President Bush has getting his nominees confirmed. 

MP does not agree that the question asked by the ABC/Washington Post poll is inherently biased: “Would you support or oppose changing Senate rules to make it easier for the Republicans to confirm Bush’s judicial nominees?”  There is much I like about this question:  It is clear, concise and easy to understand and interpret because it avoids the use of often unfamilar terms like “filibuster.” 

The problem is — and here the conservative critics have a point — it is just one question and it does reflect one particular framing of the issue.   As Ramesh Ponnuru points out, there is another question they could have asked that is equally concise and clear:  “Would you support or oppose changing Senate rules so that judges can be confirmed by majority vote?”  We might take Ponnuru’s suggestion a step further and ask whether rules “that make it easy for a minority of Senators to block a nomination even when majority of the Senate supports it?” 

Different questions may produce greater support for the Republican position, as the various results presented above imply.  Understanding public opinion with respect to judicial nominees is not about not about deciding which question is best, or whether any one question alone is biased.  It is about measuring all attitudes, even the ones that conflict, and coming to a greater understanding of what it all means.  The answers may be contradictory, but sometimes, so is public opinion.

[minor typos corrected]

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.