« September 2005 | Main | November 2005 »

October 29, 2005

Watch Out for the Easy Out

As of last week (before yesterday's indictment news) what portion of Americans believed "Bush administration officials" did something "illegal or unethical" in connection with the leaking of a CIA operative's identity?  CNN's summary of the latest CNN/USAToday/Gallup poll may have left some readers a bit confused.  A quick lesson in survey methodology may help clear up the confusion.

The CNN story leads with the following sentence:

Only one in 10 Americans said they believe Bush administration officials did nothing illegal or unethical in connection with the leaking of a CIA operative's identity.

But then three paragraphs later, the story tells us:

The poll was split nearly evenly on what respondents thought of Bush officials' ethical standards -- 51 percent saying they were excellent or good and 48 percent saying they were not good or poor.

Huh?  Fifty-one percent (51%) of Americans rate the ethical standards of Bush administration figures as "good" or "excellent," yet only 10% say they did nothing "illegal or unethical" in the Plame case.  How can that be?

One possible if unlikely explanation is that the more general question came first.  Perhaps, even before the announcement of the Libby indictment, Americans were evenly divided on the general ethical standards of Bush administration officials even though only 10% believe they did nothing "illegal or unethical" in the specific example of the Plame case. 

A better and more likely explanation has to do with the way Gallup asked the question that produced the 10% result and the way CNN chose to present it.  Let's take a look at the question:

As you may know, several members of the Bush administration have been accused of leaking to reporters the identity of a woman working for the CIA.  Which of the following statements best describes your view of top Bush administration officials in these matters - some Bush administration officials did something illegal, no Bush administration officials did anything illegal, but some officials did something unethical, or no Bush administration officials did anything seriously wrong?

39% did something illegal
39% unethical, not illegal
10% nothing wrong
12% had no opinion

This question forces a choice between three alternatives with one category (not illegal but unethical) clearly in the middle.  Survey methodologists have long debated the merits of presenting an expclit middle alternative choice in survey questions.  On the one hand, the true attitudes of the respondents may sometimes fall in the middle.  On the other, some respondents may use a middle category as an "easy out" rather than thinking hard about the issue at hand.  The academic research on this issue is mixed (see O'Muircheatough, Krosnick and Helic, 2000, for a summary), but even then the academic research MP knows is not exactly on point with this example, as it tends to focus on very concise answer categories, such as whether the government should spend "more, about the same or less" on various programs.

What the academic literature tells us more clearly is that "easy out" behavior by respondents (something methodologists call "survey satisificing") appears to occur more often when questions are wordy and confusing.  Ironically, a recent working paper based on experiments by the Gallup Organization (Holbrook, Krosnick, Moore and Tourangeau, 2005) on "order effects" (whether the order of answer categories affects the results) reached the following conclusion (pp. 22-23):

We found that response order effects were more common among questions that were more difficult to comprehend, when the response options were complete sentences instead of single words or phrases, and among questions involving response options that were not mutually exclusive. These findings are consistent with a large body of evidence from previous research that have found response order effects to be stronger for longer questions, questions with longer response options, and those with more difficult language (Bishop and Smith 2001; Payne 1949/1950; although McClendon 1986a found no such relation).

The authors of the report concluded with this advice for pollsters (p. 27) :

Our findings have a number of implications for survey research practice, though they are hardly surprising.  Because response order effects are stronger in questions that are more difficult to comprehend, researchers should strive to keep their language as simple as possible.  Advice of this sort is offered in numerous research methods textbooks, but a cursory examination of many survey questionnaires suggests that there may still be room for researchers to heed this advice more faithfully.  It is not uncommon to see questions laden with social science jargon and long sentences involving many multi-syllabic words.  Because survey researchers are usually well-educated, they may find such questions to be easily understandable. 

With all of that in mind, let's consider that Gallup question again.  Is it likely that a large number of Americans considered the actions of "Bush administration officials" in the Plame affair unethical but not illegal as of last week?  Certainly.  So posing a question with a middle category in this case seems reasonable.  However, the question they wrote is a it long and wordy side.  For example, it repeats the phrase "Bush administration officials" three times in one sentence, throwing in "officials" a forth time.  Why not simply ask whether Bush administration officials "did something illegal, did something legal but unethical or did nothing seriously wrong?"   Another complication is that the middle category poses something of a double-negative: not illegal but unethical. 

MP wonders how many respondents found this question confusing or hard to follow.  MP also wonders whether some heard the question as having four choices rather than three.  If one does not listen carefully the middle option may sound like two distinct choices:  "no Bush administration officials did anything illegal, but some officials did something unethical."  Is it possible that some respondents replied that they preferred the second option because they thought it said only that Bush officials "didn't do anything illegal?" 

MP encourages readers to try this experiment:  Read the question aloud to someone and ask them to answer it.  Better yet if that person lacks a college degree.  Are they confused?  Are they confused?  Do they ask you to read the question again?   

In a perfect world of unlimited budgets and no time constraints, survey organizations would conduct extensive pre-testing to check against such confusion.  MP has no knowledge of how carefully Gallup pre-tests its surveys, but his experience is that most pollsters lack the time and budget to pre-test as thoroughly as they would like. 

So what can we make of results of this question?  Here is what we know:

  • On another simpler question on the same survey, 51% rated the "ethical standards of top administration officials" as excellent or good, 48% as not good or poor.   
  • On the same question more than half the respondents chose one of the two middle categories: 11% rated the ethical standards excellent, 40% good, 19% not good and 29% poor. 
  • When Gallup tabulated responses to the longer question by party identification, the "did something illegal" response follows the pattern we would expect:  much higher among Democrats (55%) than Republicans (19%) with independents somewhere in the middle (45%). 
  • However, preference for the middle choice - unethical not illegal - was similar among Democrats (36%), independents (36%) and Republicans (44%).

1029_gallup_screen_shot_1


MP's admittedly subjective conclusion:  The middle category of the three way illegal/ unethical question may probably included many respondents without well formed opinions in any direction.  The 39% who believe Bush officials did something illegal is the most reliable and newsworthy result from this question, while the lines between other categories seem more blurry.  The USA Today story and the Gallup release both put appropriately greater emphasis on the 39% result.  CNN would have done better to follow their lead, rather than hyping the "only 10%" result.

Posted by Mark Blumenthal on October 29, 2005 at 03:04 PM in Polls in the News | Permalink | Comments (2)

October 25, 2005

How Low Can It Go? - Epilogue

Two days of airports and MP continues to play catch-up at the office, so just another quickie post today:

Today's streaming video "Daily Briefing" from Gallup Editor-in-Chief Frank Newport provides a helpful epilogue to our "How Low Can It Go" post from last week.  They checked available presidential approval ratings for the four other presidents in their second term and compared George W. Bush to other presidents in their "19th quarter."  The results:  At 44% approval, Bush is scoring lower than Presidents Eisenhower (60%), Reagan (61%), Clinton (59%) but better than Nixon (32%).   As always, the Gallup Daily Briefing is free to all until the stroke of midnight.   

1025_gallup_19th_qtr


Keep in mind that in the third quarter of 1973, the Nixon administration was reeling from a summer of blockbuster testimony and disclosures stemming from the Senate Watergate hearings.  However, MP's search of the Gallup archives suggests that the "19th quarter" numbers came just before the so-called "Saturday Night Massacre" in late October 1973 in which Nixon fired the Watergate Special Prosecutor and others.   

UPDATE:  As I was writing this, CNN released a new set of presidental approval numbers from Gallup. 

Posted by Mark Blumenthal on October 25, 2005 at 04:25 PM in Polls in the News | Permalink | Comments (2)

October 24, 2005

California Propositions: Pollster Showdown

MP apologizes for sparse posting over the next day or so, as his "day job" with little time for blogging.

In the meantime, readers may want to consider an emerging polling controversy in California, where pollsters have been asking about a series of ballot initiatives pushed by Governor Arnold Schwarzenegger.   Three pollsters have been tracking opinions on these initiatives, the California mega-pollsters Field and the Public Policy Institute of California (PPIC) and the national pollster SurveyUSA, that as all MP readers know, conducts surveys using an automated methodology rather than live interviewers.   SurveyUSA has been showing support for the initiatives running anywhere from 15 to 32 percentage points higher than the two California Pollsters.   As usual, the site RealClearPolitics has an excellent compilation of results and links to the pollster releases. 

Suprisingly, the proximate cause of the differences appears to come not from the mode of interviewing, but the wording of the questions.  Generally, Field and PPIC are asking relatively longer questions that attempt to replicate the language that will appear on California ballots.  Survey USA has been using shorter questions that attempt to boil each proposition down to a single sentence.   All of this is explained in some excellent reporting and analysis by William Finn Bennett in the North County Times.  Read it all.

The folks at SurveyUSA included a pollster caveat in their most recent release that discusses this issue.  In an email to MP about the issue, they also disclose that they will be asking two versions of the Prop 76 quesiton on upcoming surveys: 

As a result of the legitimate concerns raised by DiCamillo and others (both before this article was published, and in this article), SurveyUSA will begin doing split ballot question-wording on Proposition 76, so see what effect the addition of more words to the ballot measure description does to our poll results.

We'll be watching. 

Posted by Mark Blumenthal on October 24, 2005 at 12:05 PM in Measurement Issues, Polls in the News | Permalink | Comments (0)

October 19, 2005

How Low Can It Go?

Following up on yesterday's post, reader ZH asked some reasonable questions about presidential approval ratings:

How low does a president's approval rating get, realistically speaking? What is the lowest recorded rating ever?  How about in recent memory?

Back in March, the Gallup Organization's Jeffrey Jones did some analysis showing that presidential approval has sometimes fallen below 30% on their surveys:

Four presidents hit bottom below the 30% approval level -- Harry Truman (23%), Richard Nixon (24%), Jimmy Carter (28%), and the elder George Bush (29%). Four others bottomed out below 40% -- Lyndon Johnson (35%), Gerald Ford (37%), Ronald Reagan (35%), and Bill Clinton (37%). Kennedy's low point was 56%; Eisenhower's, 49%.

Most presidents' low approval ratings can be attributed to events that were demonstrably not going well for the United States -- specifically, the Korean War for Truman, the Vietnam War and racial tensions for Johnson, imminent impeachment for Nixon over the Watergate scandal, the energy crisis for Carter, and flagging economies for Ford, Reagan, and Bush the elder.

Also of note is that Johnson's and the elder Bush's low points came in polls completed just after the opposition parties held their presidential nominating conventions, which likely served to focus the nation's attention on its problems and to lay the blame for those squarely on the president's shoulders.

The Gallup analysis, which also includes complete results for the "low" for each president along with field dates, is available to paid subscribers only. 

Approval ratings for governors and other offices has been known to fall even lower.  For example, the most recent job ratings for Ohio Governor Robert Taft were 15% and 17% respectively on polls by the Columbus Dispatch and SurveyUSA.   Taft's free fall to that level occurred in part because his approval among Republicans collapsed to only 25%. 

So another way to think about how low the President's numbers could go is to consider Bush's support among Republicans.   The overwhelming majority of those that now approve of his performance are either identify as Republicans or tell pollsters they are independents who "lean" Republican (see the IPSOS crosstabs).  For Bush's numbers to fall much further, he will need to lose significant support from Republicans. 

So far, at least, any such erosion has been minimal.  The most recent survey report from the Pew Research Center included the following chart showing the trend in presidential approval over the year.

1019_pew_graph


The Pew report concluded:

The president continues to draw strong support from Republicans, 81% of whom approve of the job he is doing. But that number reflects an eight-point decline since January, with most of that drop occurring in late summer. Among independents, a plurality of 47% approved of Bush's performance in January; now just 34% do so. Approval among Democrats is now in the single digits (9%), down from 17% in January.

MP also cobbled together results by party where available from the six pollsters that released surveys in the last week.  Remember, we are looking at relatively small sample sizes (hundreds of interviews, not thousands) with more random varation, but the overall pattern is consistent with the results reported by Pew (source links:  USAToday/Gallup, Fox, Pew, WSJ/NBC, AP-IPSOS & CBS).   Bush shows an average 80% support from Republicans across the six surveys.   

1019_bush_job_by_party


MP could find only two of these organizations, other than Pew, that characterized the trend in the job approval rating by party over the last month.  Both indicate that for now at least, Bush's support from Republicans is holding at roughly 80%:

CBS:  Those in [Bush's] own party are still overwhelmingly positive about his performance (nearly 80 percent approve), but the president receives little support from either Democrats or Independents. And while views of President Bush have lately not changed much among Republicans or Democrats, his approval rating among Independents has dropped 11 points since just last month, from 40 percent to 29 percent now.

USAToday (Gallup):  Bush's fall, from a 45% approval rating in late September, is largely due to a drop in support among independents and Democrats. His approval among independents declined to 32% from 37% since the last USA TODAY/CNN/Gallup Poll taken Sept. 26-28. Approval among Democrats fell to 8% from 15%.

Bush's approval among his Republican base continues to hold firm. It was 85% in the previous poll and 84% in the latest, steady support that's preventing him from falling lower.

 

Posted by Mark Blumenthal on October 19, 2005 at 03:06 PM in Polls in the News | Permalink | Comments (8)

Diageo/Hotline Poll: Same Poll, New Name

The latest poll is out from National Journal's Hotline, although now with a new sponsor and two new names:  The Diageo/Hotline poll conducted by Financial Dynamics.  The pollster of record is still Ed Reilly, although his company Westhill Partners was recently acquired by Financial Dynamics.  So same pollster, different names. 

Their latest poll includes a parallel study on the Virginia governor's race.  At the newly renamed Diageo/Hotline poll site (update your bookmarks), readers can find  analysis, slides and full results for both the national and Virginia surveys. 

One notable result:  On an open-ended question, 33% of their national sample of registered voters (n=6500, interviewed 10/12-16) mentioned the economy as the most important issue facing the country, up sharply from 14% on the previous survey and an average of 15% on surveys conducted earlier in 2005. 

At a poll briefing that MP attended in Washington this morning, Reilly described the result as "a significant shift in perception."   He described issues like the Miers nomination, the Plame leak investigation as "abstract" in comparison to the pain of rising gas prices.  "What people are really experiencing," Reilly said, "is that money has come off the table."

UPDATE:  Reader PY writes to remind MP that 500 interviews is unusually small for a national sample and makes for small subgroup sample sizes.  For example, the party ID subgroups in the presentation include somewhere between 120 and 160 interviews each.   

Also, note that the "Candidate Profile" slide (#17) shows results as a percentage of each candidate's support.  Thus, the column for independent candidate Potts indicates that 34% of his support comes from Northern Virginia, NOT (improbably) that Potts gets 34% of the Northern Virginia vote.  As the Potts "profile" column is is based on only 5% of the vote (roughly 25 interviews), Hotline/Diageo might have done better to omit it. 

Posted by Mark Blumenthal on October 19, 2005 at 02:01 PM in Polls in the News | Permalink | Comments (1)

October 18, 2005

Getting Past the Noise: Bush Slide Continues

The release of new results from six major public pollsters over the last two weeks provides a great opportunity to learn something about both the limitations of survey research and about its power. 

Most political surveys are based on random samples of telephone households, which comes with a lot of random variation.  As we have seen in two recent posts, very small sample sizes (100 interviews or less) can show huge variation, but even larger samples (500 to 1000 interviews) still show a lot of statistical noise.  Remember that the +/-3% "margin of error" on 1000 interviews assumes a 95% confidence level.  That means that one survey statistic in twenty will fall outside that [of] range [of the "true" value] by chance alone.  And none of this considers the variation that can come from other polling variables:  response rates, question language and order, interviewer characteristics and training, pollster persistence in contacting the respondents, techniques used to select respondents and so on. 

The bottom line:  Political polls come with a lot of random noise.  Anyone poll may show a difference of a few points when compared to another that turns out to be completely meaningless.  Unfortunately, all too often such variation gets reported as a real trend. 

However, the power of such polls is that the more we can increase the size of random samples, and the more we can look at not just one poll but many polls, we can begin to see real trends emerge from all the random noise.

Such is the result of the six polls released in the last two weeks, including one released this morning by Gallup/CNN/USAToday.   The table that follows compares results from six national polls released in the last two weeks to the average of results from the same pollsters as measured in September.  All six showed the job approval rating of President George W. Bush declining since September. 

1018_all_job_ratings


Now, the margin of error allows for the possibility that any one survey might show Bush a little higher or a little lower by chance alone.  If perceptions of Bush's performance had not changed since September, we might expect to see some polls showing him doing better, some doing worse.  The mix should be roughly half up, half down, as if we had flipped a coin six times.  But in this case all six show a decline.  My favorite "binomial calculator" tells me that the probability of flipping a coin six times and having it come up tails each time is only 1.5%.  (For more details about the various polls and links, see the PollingReport or RealClearPolitics). 

An even better way to separate trends from noise is to draw a picture.  Or more specifically, plot a graphic that shows every job approval rating released on a national poll during 2005.  Prof. Charles Franklin of Political Arithmetik did just that last week.  This is what the picture looks like (the red dots are polls conducted since Katrina);

1018_franklin_job_approval



Franklin plotted a regression line through the dots that shows the trend embodied in all the polls after correcting for "house effects" - consistent "differences among polling houses" (as he puts it).**  Here is Franklin's take:

The decline in the President's approval is clear visually, and the local trend line in red shows how sharply the trend has departed from the blue linear trend for PRE-Katrina polls (Note the close fit of red and blue trends before August.)

The model, corrected for differences among polling houses, now estimates a Katrina effect of -1.2%. Prior to Katrina, approval was falling at a rate of -.03% per day throughout 2005....

The bottom line: the President's approval has fallen all year, declining about 1% every month since January. But since August we've seen a sharper drop. [emphasis added]

In the chart you can literally see the random noise, the roughly 10 percentage point band of dots around the trend line.  That is a picture of the limits of survey research.  But we can also see a clear, unmistakable trend.  So it is also a picture of the power of polling. 

**See the non-self-aggrandizing page of R. Chung for some incredible graphics on public pollster house effects.

UPDATE:  This afternoon SurveyUSA released it's monthly 50-state tracking of the Bush job rating.   The also compute a weighted overall average with each state weighted to its share of the US population.  The result is consistent with the other pollsters:  From 41% approve-57% disapprove in September to 38% approve-59% disapprove now. 

Posted by Mark Blumenthal on October 18, 2005 at 03:05 PM in Polls in the News, Sampling Error | Permalink | Comments (5)

October 15, 2005

UPDATE: 2% Among African-Americans?

A quick update on yesterday's post:  First, some of those who left comments yesterday seem to miss the main point.  MP has no doubt that African-Americans express nearly monolithic disapproval of President Bush, far more than any other demographic subgroup.  The question is whether the small sample of African Americans in this week's NBC/Wall Street Journal poll justified the attention and "free fall" rhetoric it inspired.

MP contacted several public pollsters that also released national surveys in the last week.  They were hesitant to release results from their small subsamples of of African-Americans because they rarely report on subgroups of less than 100 interviews.  Here are two responses:

Kathy Frankovic of CBS News:

We have on a few occasions reported subgroups of less than 100 respondents.  Usually that would be to make the point that there was nothing unusual about the group -- or that there was something very different about them (blacks on whether the slow response to Hurricane Katrina was motivated by race, for example).  But those are exceptions -- usually we don't even look when we know the sample size will be small.

Dana Blanton Stanton of Fox News:

When we get a request for a small subgroup, we try use characterizations rather than exact percentages.  If pushed to use a number, we prefer to use a rolling average of three polls instead of just one poll

MP is sympathetic.  The main point of yesterday's item was to warn of the dangers (and questionable news value) of very small sample sizes.  So pressuring the pollsters to release data from similarly small recent samples of African-Americans seems a bit hypocritical.   

So rather than focus on their polls from last week, MP asked if they would be willing to roll together their recent samples to increase the sample sizes and tabulate the Bush job rating for two periods: Polls conduced since Katrina (from September to October) and those conducted earlier in 2005 (from January to August).  Here are the results.

1015_bush_by_race


Some important notes:  The CBS and Pew Research Center surveys -- like the NBC/Wall Street Journal survey -- interview national samples of adults.  The Fox News/Opinion Dynamics conducts a survey of self-described registered voters only.  Second, both CBS and Pew did additional "over-samples" of African Americans on their first post-Katrina studies in September.  Also, the the number of interviews ("n=") provided for the Pew and CBS surveys are unweighted totals.   Finally, the omission of results for white voters by CBS is MP's doing.  He neglected to ask. 

So what do these numbers tell us?  With larger sample sizes, neither the CBS nor the Pew surveys show much change in Bush's job approval among African-American voters since Katrina, although the view of Bush among black voters has obviously been very negative all along. 

On the other hand, the Fox News survey of registered voters  shows a big drop (from 22% to 7%) in Bush's job rating among African-Americans.  Whether this results from the difference between all adults and registered voters or some other factor is unclear. 

In any event, these numbers strongly suggest that the 2% statistic understates Bush's job rating among African Americans.  The pollsters from Fox and CBS were willing to characterize their most recent samples as consistent with their recent polls conducted from September through October.  So if we were to average the results among African Americans on all the polls released in the last week, the Bush job approval rating would probably be at about about 10%. 

Why does this matter?  Isn't 10% a pretty low number?  Yes, but consider it this way:  If the NBC/WSJ poll had estimated Bush's approval among African-Americans at 10% rather than 2%, would Tim Russert have gotten as excited ("2 percent!") and reported the number as prominently on the Nightly News?  Would the Washington Post's Dan Froomkin have devoted a column to what "what may turn out to be one of the biggest free-falls in the history of presidential polling?"  Would the Huffington Post have hyped the statistic with a banner front page headline?  Would Maureen Dowd have included it in her (subscription required) column today?

MP doubts it.   

And if you still find the 2% statistic "plausible," ask yourself this question:  If the next NBC/WSJ journal poll shows Bush's approval rating at 10%, should Russert take to the airwaves to report a dramatic "rebound?"

LATE UPDATE:   Looks like Russert continued to tout the 2% figure on Sunday's Meet the Press (thanks to reader KW).

Posted by Mark Blumenthal on October 15, 2005 at 02:19 PM in Polls in the News | Permalink | Comments (3)

October 14, 2005

2% Among African-Americans?

Has George Bush's job approval among African Americans really fallen to just 2%?  While MP was off atoning for his sins, several national pollsters released new surveys.  More on the bigger picture later, but for now let's focus on that 2% number that comes from the latest Wall Street Journal/NBC poll.  While this particular result appeared nowhere in either of the poll summary stories that appeared in the Journal or on MSNBC.com, Tim Russert hyped it with great enthusiasm to a considerably larger audience on the NBC Nightly News with Brian Williams on Wednesday night:

Brian, listen to this.  Only 2 percent -- 2 percent! -- of African-Americans approve of George Bush's handling of the presidency -- the lowest we have ever seen in that particular measure. 

WashingtonPost.com's Dan Froomkin (who also transcribed the above) contacted Democratic pollster Peter Hart, who regularly conducts the NBC/WSJ poll with Republican Bill McInturff .  That support, according to Froomkin, registered at 19% "as recently as six months ago."  Hart, Froomkin reports, had "never seen such a dramatic drop in presidential approval ratings, within any subgroup."

Now, the full results posted online by the Wall Street Journal do not include either a cross-tabulation of the job rating by race or the racial composition of the poll sample.  However, Froomkin reports the following:

This latest poll included 807 people nationwide, and only 89 blacks. As a result, there is a considerable margin or error -- and the findings should not be considered definitive until or unless they are validated by other polls. 

How big is that margin of error?  Assuming simple random sampling, the standard 95% confidence level used in most public opinion surveys, the maximum margin of error on 89 interviews is  10%.  However, those who took statistics may remember that the maximum error range applies for percentages near 50%, it gets smaller as percentages approach 0% or 100%.  If one plugs 2% and n=89 into the standard formula for the margin of error, and I get +/- 3%.  However, statisticians have long debated whether we should ignore the sharp reductions in the margin of error at very small percentages.  Regardless, media pollsters are typically cautious about reporting on results from sample sizes of less than 100 interviews.

That is one reason why national media pollsters typically conduct special "oversamples" of African Americans when they want to investigate racial differences in attitudes.  They weight those additional interviews back to match census estimates of the racial composition of American adults for their overall tabulations.  However, the extra interviews give them more confidence in crosstabulations by race.  Gallup, CBS/New York Times and ABC/Washington Post did just that with their post-Katrina polls last month. 

MP also wonders about the 89 interviews of African Americans that Froomkin reports.  Is that a weighted or unweighted total (and thanks to a loyal but anonymous MP reader that emailed with that question)? MP suspects it is weighted and that the actual number of African American interviews was a smaller number than 89. 

Here's why:  89 interviews amounts to 11% of a sample of 807 adults.   Eleven percent happens to be the percentage of African-Americans among US adults (as estimated by the 2000 Census).  Most national pollsters weight their samples to match that estimate (see the percentage black in the last Washington Post poll).  Response rates tend to be lower in urban areas, and as a result, unweighted national samples typically under-represent African Americans.  As such, most national pollsters weight African-Americans up slightly to match the estimates provided by the US Census.  Thus, the odds are good that the WSJ/NBC poll did not come in at 11% black unweighted. 

But MP is burying the lede.  Rather than speculate, we can simply check this result against other national polls conducted in the last week to see if the NBC/WSJ poll was a statistical outlier.  Dan Froomkin found one:

Late Update: The Pew Research Center is just out with its latest poll, which has a larger sample, and it finds Bush's approval rating among blacks at 12 percent, down only slightly from 14 in July. Here are those results.

Note:  Pew's sample included 135 African Americans and was fielded October 6 through 10.   It should not be a difficult matter to obtain similar tabulations from some of the other public polls released this week.  So stay tuned.

Meanwhile, a note to powerful pundits who present polling data on national television:  A sample of less than 100 interviews can be a dangerous thing.  Handle with caution, not exuberance.

PS/Irony Watch Update:   Few have been more consistently derisive of political polling than Arianna Huffington.  In 1998, in a post titled “Margin of Error or Credibility Gap,” she wrote:

Isn't it time we took pollsters at their word and started treating polling results as estimates? Wouldn't it be nice to see these "snapshots" posted in the back of the newspaper, next to the astrology charts?

Looks like the Huffington Post isn’t exactly following that advice. As of this writing, the 2% among African Americans result blares from its front page.  The HuffPo “Newswire” says [correction: quotes from Dan Froomkin's lead]:  The result “may turn out to be one of the biggest free-falls in the history of presidential polling.”

Uh huh.  Perhaps not as “amazing” as the first item on the HuffPo front page (see the screenshot) but still quite amazing.

1014_huffpo


UPDATE:  I've posted some additional data and thoughts in a new post - continue reading here

Posted by Mark Blumenthal on October 14, 2005 at 01:48 AM in Polls in the News | Permalink | Comments (19)

October 07, 2005

SurveyUSA on Miers

MP somehow overlooked that Survey USA is now running two separate automated tracking polls on the Miers nomination (see previous commentary on their recorded voice methodology).  One track has been following Harriet Miers’ favorable rating since last Saturday, when rumors of her nomination first started to swirl (see summary, crosstabs & tracking graph).  They have also just released a second survey conducted yesterday that asks some questions you won’t likely see asked by other pollsters (some of which are a bit leading): Will the Senate confirm Miers (not should they)?  Does Miers have “the potential to go down in history as one of the Supreme Court's greatest legal minds?”   Will the Miers nomination be seen as “one of President Bush's most brilliant decisions” or as one of his “greatest mistakes?” And so on (see summary & crosstabs). 

One intriguing thing about the SurveyUSA favorable rating is that the results from Monday and Wednesday are roughly comparable to the results obtained by the CBS poll conducted Monday through Wednesday evenings.  CBS had 11% rating her favorably, 11 unfavorably and the rest (76%) either undecided or had not “heard enough to have an opinion.”  On October 3, SurveyUSA showed 14% rating her favorably, 9% unfavorably and 77% had “no impression.”  On the other hand, they do show a jump in awareness of Miers on Wednesday night, with 28% favorable, 13% unfavorable and 59% with no impression.  Average the Monday and Wednesday interviews for SurveyUSA, and we get 68% for “no impression.” Given that it is probably easier to say one has not “heard enough to have an opinion” than to say one has “no impression of her,” the CBS and SurveyUSA numbers look well… not inconsistent.

Also, at risk of thinking out loud, MP wonders whether the one night SurveyUSA polls may produce samples that are slightly more attentive and informed than those conducted over three nights.  Any one night methodology will tend to overrepresent those who are always home and underrepresent those who tend to spend less time at home.  Would a one-night sample conducted on Wednesday night tend to overrepresent Americans who watched the news that night?  MP guesses it might. 

107_susa_miers


All of this is worth pondering because the SurveyUSA track shows a sharp upward trend in awareness of Miers (that translates into increased favorable and unfavorable ratings) over the last four days (see/click the chart above) which suggests that the CBS survey may already be a bit dated. 

MP assumes we will have many more such surveys in the days and weeks ahead.  It will be interesting to watch.  Stay tuned. 

Posted by Mark Blumenthal on October 7, 2005 at 03:04 PM in Polls in the News | Permalink | Comments (0)

CBS on Miers

Last night, CBS News released a new and very comprehensive survey conducted Monday through Wednesday evenings this week (The CBS web site has complete results plus a two-part written analysis covering the Miers nomination, the Bush job rating and more).  It includes some data that begins to provide a hint at the historical context in regards to the Harriet Miers nomination that our friend Mickey Kaus has been looking for (scroll to "First Poll")

There is much in this survey to chew over (including the finding of a "new low" in the overall Bush job rating), but for now, MP will just pass on some of their findings on the Harriet Miers nomination along with some of their historical numbers.

One thing that CBS often does a bit differently than other pollsters is to prompt respondents to say when they "haven't heard enough to have an opinion."  They have always done so on questions about Supreme Court nominees.  As such, this practice probably gives us a better sense of those with truly informed opinions (as opposed to respondents who form an opinion on the spot in the midst of the survey). 

On their latest poll, CBS found an overwhelming majority of Americans either "undecided" about Miers (18%) or who haven't heard enough to say (58%).  The rest split evenly, with 11% rating her favorably and 11% unfavorably. Earlier in the summer, a full ten days after his nomination to be associate justice, CBS found 25% rating John Roberts favorably and 7% unfavorably.  As with the Gallup survey released earlier this week, CBS found the biggest differences in reactions to Miers and Roberts among Republicans and conservatives.

However, CBS also asked the ultimately more pertinent question, one they have asked about previous nomineees: "Should the Senate vote to confirm Harriet Miers as a Justice of the U.S. Supreme Court, or vote against Miers, or can't you say?"  Most (70%) could not say, but those with an opinion split almost evenly, 14% to confirm and 13% to reject.

Earlier this month, in the context of a poll on John Roberts, CBS provided results of this question when asked about Robert Bork (in 1987) and Clarence Thomas (in 1991). To be fair, the historical data comes later in the confirmation battles of the earlier nominees, all of whom had been announced several months before CBS asked the "should the Senate confirm" question.  Obviously only a few days have passed since Bush the Miers' selection, and opinions about her may well change in the coming weeks and months. 

With that important caveat in mind, here is how the comparisons stack up (the field dates for the surveys, as provided by CBS, are in parentheses):

Cbs_supremes_1


Again,  it's early.  However, these results show that most Americans remain inattentive to the Supreme Court nominees even toward the end of high profile fights like those over the Thomas and Bork nominations.  Considering that, the fact that early reactions for Miers look more like Bork than Thomas or Roberts does not bode well. 

Posted by Mark Blumenthal on October 7, 2005 at 02:34 AM in Polls in the News | Permalink | Comments (9)