Another “New Low?”

In his Washington Post “Media Notes” blog, Howard Kurtz raises an important question about the coverage of the polls showing “new lows” for George Bush (hat tip: Kaus).  Kurtz and Kaus are right to wonder if the “new low” headlines give a distorted impression of a continuing day-to-day decline and right to imply that when it comes to covering polls, more references would be better than few. 

The question Kurtz asks at the end of this passage is definitely worth considering: 

Can I just grumble a little about this USA Today /CNN poll?
“President Bush’s ‘approval rating’ has sunk to a new low according to a USA TODAY/CNN/Gallup poll released Monday.

“The latest results show only 36% of those polled saying they ‘approve’ of the way Bush is handling his job. Bush’s previous low was 37%, set last November.

“Sixty percent of those polled said they ‘disapprove’ of Bush’s performance. That matches an all-time worst rating hit last November and again two weeks ago.”

Bush is at a new low compared to USA’s last poll. CBS has Bush at a new low compared to the last CBS poll. Etc., etc. All true, but they give the collective impression that Bush is sinking week to week. Why do they only compar[e] figures to their own past surveys, when they’re fully aware of the others?

One good reason to avoid cross-pollster comparisons is the issue of “house effects.”  Some pollsters typically show Bush a little higher than others, some a little lower.  These effects are not evidence of intentional “bias,” but rather due to small differences in methodology.  Some are obvious (such as the Fox poll interviewing only registered voters) some less so (such as why CBS, Fox and the Pew Research Center seem to get a higher “don’t know” response on the job rating question than other pollsters).   See the posts on “house effects” by Charles Franklin and especially Robert Chung.

But the potential for spurious cross-pollster comparisons does not entirely explain why news organizations shy away from references to polls other than their own.  They could do what we do here at MP: make apples-to-apples comparisons across a number of different polls. 

So let’s take a moment and do just that, with an assist with a graphic specially created for MP readers by our friend, Professor Charles (“Political Arithmetik“) Franklin. The chart below [click it for a full size version] shows the overall Bush job approval rating on every national public poll released since Labor Day, with lines plotted for seven pollsters.

Three important points emerge from the chart: 

• Every survey released since February 19 (the grey vertical line) shows a decline of at least four percentage points since early January, with every pollster except ABC/Washington Post showing that decline occurring during February.   
• The lows of late February roughly match those recorded in later October.  Collectively these levels represent the lowest measured during the Bush administration (as confirmed by another graph on Political Arithmetik). 
• Over the last three weeks, only Gallup, CBS and automated pollster Rasmussen have fielded more than one survey.  None show any statistically significant change since during that time. 

Which brings us back to the important part of Kurtz’ question:  Why does each intra-pollster comparison herald a new “new low” headline, when the more accurate read (for now) would be to treat each new survey as confirming the same drop since February recorded by all?  Such an approach does not require a snazzy graphic like the one above (though it couldn’t hurt). It could be done in a sentence:  “Our results are consistent with those last week by …”

Actually, the very article that Kurtz grumbled about included just such a reference:

[The latest Gallup] results closely resemble those reported last week in an Associated Press/Ipsos poll. That survey put Bush’s approval rating at 37% – matching the AP/Ipsos poll’s all-time low.

However, Kurtz is certainly right that such references are rare. I can think of three reasons for that reticence, and all three should give both pollsters and journalists pause. 

First, the “new low” headline or lead is a lot more dramatic and easier to write.  Most pollsters are guilty of this habit – I know I have done it in internal memos to clients – even when we hasten to add in the next sentence that, well, the new low is only a point or so lower than the old low measured last week, and therefore isn’t statistically significant.  But still, it is a new low!  This bad habit is a matter of human nature.  No pollster wants to report that the latest survey reveals “no significant change.” 

Second, news organizations consider it bad form to give prominent play to surveys done by their competitors, even when as Kurtz points out, “they’re fully aware of the others.”
Ironically, in a recent online chat posted elsewhere on WashingtonPost.com, a reader asked Post polling director Richard Morin a similar question: 

Albany, N.Y.: Do you believe that the polls done by other news outlets are just as newsworthy as your own poll, and if so, why do they not receive the same degree of prominence in The Post?

Richard Morin: No, frankly, I don’t, and for the same reason that we do not publish stories about the same press conference done by USA Today, Wall Street Journal, The Los Angeles Times, that other Times, etc. We run our own. That does not mean we do not cite other poll findings, or that we won’t give strong play to a poll on a topic we haven’t surveyed on. It’s just that we have more confidence and more control over our own polling.

Morin, to his credit, did use the same chat session to compare and contrast the most recent Post/ABC poll results to other recent surveys by Gallup, Diageo/Hotline and CBS/New York Times. 

Third, there is the issue of marketing.  As Tom Rosentiel put it in his recent POQ article, “Political Polling and the New Media Culture” (2005, 69:5, p. 703), “marketing considerations” drive the way polls are used, “even at major news organizations.”  Rosentiel related a fourteen-year-old story about a strategic planning retreat at the Los Angeles Times, in which an unnamed editor discussed the Times poll:

The paper would be conducting this many polls and spending this much, he pointed out, as if to cut off any further discussion on the point, because we needed to recognize something. Polls are a form of marketing for news organizations. Every time a Los Angeles Times poll is referred to in other media, the paper is getting invaluable positive marketing of its name and its credibility. The same was true for any news organization, CBS News, the New York Times, the Wall Street Journal, and the rest. So if there was more money devoted to this than the assembled political journalists thought necessary, that was the reason, he was implying, and there was no point in arguing the issue.

None of these issues should preclude reporting that references polls sponsored by multiple news organizations.  The marketing considerations should be generally enhanced if more stories made reference to more polls, and even Richard Morin’s misgivings about giving prominent play to other polls obviously did not get in the way of his referencing those polls on WashingtonPost.com.  Having said that, I am not sure I agree with his press conference analogy.  We certainly would not expect the Washington Post to reprint press conference stories by other newspapers, but in that case the specific underlying facts of the event, the words spoken by the president, are not in question.  All reporters work from the same recording or transcript. 

Polls are different.  They may all attempt to measure the same underlying “public opinion” on the same issues, but the sampling methodologies and question wording used add fuzziness and uncertainty to each measurement.  Thus, the more polls you look at — whether in a graphic like the one above or a listing of questions using different language to ask about the same issue — the clearer the picture becomes.  With polling, more is better.