Mystery Pollster

Another Day, A Very Different Result

Well, another day, another -- very different -- set of poll results on the NSA phone records surveillance issue.  On a new survey out today from Newsweek (story, results) 53% of Americans said the "program goes too far in invading people's privacy" while 41% agreed with the alternative statement, "it is a necessary tool to combat terrorism."  A similar question on the ABC/Washington Post released yesterday showed 63% endorsing the NSA program as "an acceptable way to fight terrorism," 35% found it unacceptable.  So what gives?

The Post/ABC poll, conducted entirely on Thursday night, sampled 502 adults nationwide, with a margin of error of  +/- 4.5%.  The Newsweek poll was conducted over two nights -- Thursday and Friday -- and sampled 1,007 adults, with sampling error reported as +/- 4%.  Clearly, sampling error alone does not explain the difference.

What about the extra night of interviewing?  A story posted by Editor and Publisher today concluded: 

Most likely views changed that much in one day after more negative media reports (including many from conservative commentators such as MSNBC's Joe Scarborough) surfaced. The Washington Post survey took place before many Americans had heard about, or thought about, the implications.

While attitudes on this issue are undoubtedly evolving as Americans learn more about it, I am skeptical that attitudes shifted quite so dramatically in a single evening.  Given the overlap of interviews on Thursday night, an enormous change would have been necessary to explain the difference between the two polls.  While two-night polls like the one Newsweek does are not designed to track day-to-day changes, a change that big between Thursday and Friday would have been hard to miss.  If the two nights showed different results, the Newsweek story makes no mention of it.

Also, if criticism from prominent conservatives fueled any such shift, as per the Editor and Publisher speculation, I would expect to see the biggest differences between the Newsweek and ABC/Post poll occur among Republicans.  Yet as the table below shows, the difference was mostly constant across the party subgroups.  If anything, the difference was slightly greater among Democrats.

More often than not, the text of the questions is the culprit for this sort of difference.  But compare the two questions below.  They are remarkably similar, at least at first glance:

ABC/Washington Post :  It's been reported that the National Security Agency has been collecting the phone call records of tens of millions of Americans. It then analyzes calling patterns in an effort to identify possible terrorism suspects, without listening to or recording the conversations. Would you consider this an acceptable or unacceptable way for the federal government to investigate terrorism? Do you feel that way strongly or somewhat?

63% acceptable
35% unacceptable
2% no opinion

Newsweek:  Now on another subject . . . As you may know, there are reports that the NSA, a government intelligence agency, has been collecting the phone call records of Americans.  The agency doesn't actually listen to the calls but logs in nearly every phone number to create a database of calls made within the United States.  Which of the following comes CLOSER to your own view of this domestic surveillance program:

41% It is a necessary tool to combat terrorism
53% It goes too far in invading people's privacy
6% no answer

The two questions are not identical, of course.  "Acceptable," as posed by ABC/Washington Post may have been an easier test to pass than "necessary" as posed by Newsweek.  The Newsweek version also makes the "unacceptable" side of the choice a bit easier ("goes too far in invading people's privacy")  and describes the program as involving "domestic surveillance."  But had I seen both questions in advance I would not have guessed these minor variations in wording would have produced such large differences. 

A better [Another] explanation may be the context in which the questions were asked.  The Post/ABC poll involved just six questions, all of which touched on the issues of terrorism, privacy and the NSA phone records search.  The question above was asked fourth, probably within the first minute or two of the call.  [The three questions on the NSA phone records issue] followed immediately after a forced choice question about the trade-offs between protecting privacy and the investigation of "terrorist threats."

[Correction:  I overlooked the question numbering on the ABC/Washington Post release.  The three questions specific to the NSA phone records issue were numbered 44 through 47, so the ABC/Post poll was probably of the same length as the Newsweek poll with both sets of questions following roughly the same number of questions.  I obviously do not know for sure, but it appears that the ABC/Post poll went into the field on Thursday night on a poll that remains in the field as of this writing].

The Newsweek poll involved more than 40 items and asked the NSA question at the end of the interview, after eight to ten minutes spent asking about other topics (that's an educated guess based on the number of items).  These included questions yielding sharply negative job approval ratings for the President on issues like the economy (59% disapprove), Iraq (62%), health care (62%), Social Security (60%), the budget deficit (70%), immigration (61%) and gas prices (76%).  The Newsweek pollsters also asked an unusual question of the 48% who said they thought President Bush's job performance had "gotten worse" in office.  On Q11, they read a list of ten possible reasons why their opinion may have soured on Bush - including "the warrantless wiretapping that Bush authorized" and the criminal charges against Tom Delay, Jack Abramoff and Scooter Libby - and asked respondents to identify which were important in shaping their opinions. 

So in short, the ABC/Post poll asked its NSA question right out of the box, following three other questions on terrorism and privacy, while the Newsweek question followed a long review of the full gamut of issues, with an extra reminder of the various negative stories about the Bush administration for those already inclined to dislike him.  Call it "priming" or "framing" or whatever you will, but the context in which the NSA questions were asked on these two polls was wildly different.

[Correction:  It appears that both surveys asked the NSA questions after a long review of other  issues, although we do not yet know the other questions asked on the ABC/Post poll.  The context of the key NSA records questions on the poll were still different, but not nearly as different as I had initially assumed.    The ABC/Post question followed two others on privacy and the tradeoff between privacy and the investigation of "terror threats."   The Newsweek poll's first question on the NSA phone records issue was the one reproduced above.  Question order effects -- or "priming" or "framing" -- may have played a role in these differences, although the difference in context may not be quite as stark as I had initially assumed].

Obviously, I am speculating.  And yes, as reader Tano put it in a comment yesterday, questions like these probably tell us about more about "gut-reaction, (or more likely, no informed reaction at all)" than about "fully formed opinion."  But we need to remember that on complex public policy issues like these, a large portion of the population will remain inattentive and ill informed.  Many will never "fully form" opinions about these sorts of issues.  Yet "gut reactions" to arguments and counterarguments in the midst of political campaigns are the quasi-attitudes that often drive vote choice.   

When public opinion polls probe reactions to complex policy issues, they sometimes get conflicting results because differences in language and context push those without well formed opinions in different directions.  For that reason, the "best question" is rarely just one question or poll.  The most "scientific" approach is to look at many different polls that ask about the same issue in different ways and compare the results (see Professor M's advice).  In that regard, I would have done better yesterday to heed the comment of reader Nadia Hassan: "I guess we'll just have to see more polling for a clearer picture."  Over the next several weeks, we will see many new polls asking about this issue, and this confusing picture should grow clearer with each new survey.

If one of these two polls we have now turns out to be a true outlier, we will know soon enough.  Meanwhile, the safest bet is that reality of public opinion at the moment falls somewhere in between.

UPDATE (5/15):  USAToday/Gallup released a new survey this morning that goes into far more depth on this issue and helps explain the conflict between the first two surveys -- see my update here. 

ABC/WaPo on NSA Phone Records

Did yesterday's USAToday story on the collection of domestic telephone records by the NSA do President George Bush a backhanded favor?  An overnight survey released earlier this morning by ABC News and the Washington Post suggests that it may be doing just that.  Here's the lead from the Post's Richard Morin:

A majority of Americans initially support a controversial National Security Agency program to collect information on telephone calls made in the United States in an effort to identify and investigate potential terrorist threats, according to a Washington Post-ABC News poll.

The new survey found that 63 percent of Americans said they found the NSA program to be an acceptable way to investigate terrorism, including 44 percent who strongly endorsed the effort. Another 35 percent said the program was unacceptable, which included 24 percent who strongly objected to it. A slightly larger majority--66 percent--said they would not be bothered if NSA collected records of personal calls they had made, the poll found.

Complete results and the text of the questions are available in this PDF file also released this morning by ABC News. 

Yes, this survey involves a relatively small sample size (502 adults) with a slightly larger margin of error than other national polls (+/- 4.5%).  And yes, calling was completed in a single evening, a practice would have missed those not at home last night and possibly overrepresented those watching the news on television last night.   However, the findings are generally consistent with previous polling on the NSA domestic eavesdropping. It is also worth remembering that the brief upward movement in President Bush's job approval rating coincided with the disclosure of the stories on NSA wiretaps in January.

MP makes no predictions, but Bush can only stand to gain if the public's attention shifts from his handling of gas prices, the economy, immigration and Iraq to his administration's efforts to "investigate terrorism."  The Post-ABC poll found that 51% approve (and 47% disapprove) of "the way Bush is handling Protecting Americans' privacy rights as the government investigates terrorism."  That is "hardly a robust rating," as the ABC release puts it, "but one that's far better than his overall job approval, in the low 30s in recent polls."

UPDATE:  Reader DRM posts a comment below worth considering:   

I'm disappointed you didn't discuss my belief that the results of a telephone survey about privacy and electronic monitoring are likely invalid by definition. A person who places a low value on their personal privacy seems to be much more likely to speak on the telephone and answer personal and potentially controversial questions than a person who places a high value on personal privacy. I'm not sure you could pick a question for a phone survey that would have a significantly greater skewing effect than the Post/ABC question.

I'm not sure questions about privacy are "invalid by definition," but DRM makes a fair point about the potential for non-response bias on a question like this.  In other words, those who refuse the survey may have different opinions on these issue than those who participate.

It certainly makes intuitive sense that those who refuse interview requests are more likely to value their privacy, and thus be more likely to take the privacy-protection side of this debate.  But ultimately the extent of any such "skewing" is unknowable, since hard core survey-refusers would presumably refuse to participate whether the survey was done on the phone, in person, by mail or over the Internet.   

One helpful though admittedly inconclusive bit of evidence comes from the Pew Research Center study from 2003 that compared findings from two parallel surveys:  A "standard" five-day survey that expended "the same amount of effort that would be applied to any Pew Survey Project" had a response rate of 27% (using (using the AAPOR3 definition).  A second "rigorous" survey obtained a response rate of 51% with a much longer field period and greater effort to reassure respondents of the legitimacy of the project (procedures that included advance letters, monetary incentives and follow-up letters to refusals to "describe the survey process" -- see page 13).   

Both surveys included the following question:  "How much do you worry that computers and technology are being used to invade your privacy?"  The responses were virtually identical on both surveys.  Thirty-nine percent (39%) of standard survey respondents and 37% of the rigorous respondents said they worried "a lot;"  exactly the same percentage on both surveys (69%) worried either "some" or "a lot." 

Two conclusions:  First, even among the more eager respondents, respondents were not shy about expressing concerns about invasions of their privacy to a stranger on the telephone.   Second, the initially reluctant respondents expressed no greater privacy concerns than those who were initially more eager to do the survey.   Of course, these data do not preclude the possibility that the roughly 40% that refused might have different feelings about the invasion of their privacy, but my take from all of the data is that any "skew" is probably modest.   

Of course, your conclusions may differ.  And even anonymous comments are welcome below. 

UPDATE (5/13):  The results of the Newsweek poll, released on Saturday look very different.  See my take here.   

Mom's Pay: Another Non-Random Sample

The Wall Street Journal Online's Carl Bialik is on the case of another non-random sample survey conducted online.  In this week's Numbers Guy column (free to all) Bialik writes about a study done by the salary compensation firm salary.com that attempts to estimate how much it would cost to replace a stay-at-home mom with salaried professionals.  While Bialik focuses mostly on the questionable ways this study attempts to "valuate the 'mom job,'" as Salary.com puts it, readers should not overlook the non-random way they collected the data used to make those estimates.

Bialik tells us that Salary.com conducted an online survey that asked 400 mothers about "how many hours they spend on each job function each day."  How did Salary.com select those 400 women?  You won't find the answer on the Salary.com online release, but Bialik tells us that to get these 400 interviews, "the company emailed 30,000 people who use its Web site."

That's a long way from a random sample of anything.  At best, it may represent women who registered to use Salary.com - it cannot possibly represent all mothers nationwide with any statistical precision.  Now in fairness, this "study" differs from the other non-random internet survey that Bialik looked at recently, in that it makes no explicit claim of being "scientific" or possessing a "margin of error."  However, Salary.com was more than willing to make the implicit claim that their study represents all mothers nationwide. 

Check the "facts" they highlight from the "Mom Salary Study." Salary.com is perfectly happy to use their 400 non-random interviews to project statistics about "moms" in general:

"Working moms get less sleep.  Working Moms reported getting only 6.4 hours of sleep per night, versus 6.7 for the Stay at Home Moms."

"Working Moms and Stay at Home moms both spend roughly 4 hours per week nurturing the emotional needs of their kids in the "mom job" of psychologist."

"Moms work an average of 90 hours a week whether they are a Working Mom or a Stay at Home Mom"

And so on. 

This criticism may seem like overkill given the questionable methodology Salary.com used to translate these time reports into salary estimates (see Bialik's piece), but it involves the most important bedrock principle of survey research.  In order to use a "survey" of 400 people to estimate the characteristics of 30 million or more, you need to start with a random sample. 

That may seem obvious, but it's a principle that a number of otherwise smart reporters seem to have missed.

PS:  An opinion piece by Wendy McElroy of Fox News yesterday also criticized the Salary.com study yesterday, although her column included this odd bit of information:

Salary.com's press release cleared up one issue. The site had conducted a survey, not a study, as the majority of the media reported. A study is a scholarly or scientific investigation that uses controls to prevent bias and error. A statistical survey collects data by interviewing or asking questions of individuals. A survey is less rigorous but, depending on its methodology, it can produce valuable results.

Um..really?  MP is not familiar with that particular distinction (though we heard something similar recently).  The last time I checked, "statistical survey" implies a random sample that is projective of some larger population.

By the way, McElroy also takes issue with the fact that sample of stay-at-home mothers is tiny fraction (".00357 percent") of all stay-at-home moms.  Again, the basic principle of survey research:  The issue is not the size of the sample, but the way it's selected.  A sample of two hundred can be used to make estimates of a 5.6 million (within a predictable range of statistical sampling error) provided that the sample is chosen at random.   This one was not. 

The Latest from CBS/NYT

Another day, another very high profile poll release.  Today, it's the latest from the CBS/New York Times (see NYTimes article, results; CBS article, results, Medicare results).   A few quick thoughts at the end of another busy day:

• The true poll junkie will want to flip back and forth between the PDF releases from CBS News and the New York Times.  As always, the release from the Times has complete time series data -- that is, results on comparable questions on every CBS/NYTimes poll (and many that were done just by CBS) going back ten year or more in some cases.  As always, the two CBS releases provide results for every question in the survey broken out among Democrats, Republicans and independents.

• Like the USAToday/Gallup survey earlier this week, this latest CBS/New York Times survey seems to be getting more attention from the left wing of the blogosphere than the right.  I discovered the new "buzz tracker" feature on Real Clear Politics last night (after a hundred or so RCP readers used it to click through to yesterday's post).  They are currently listing 28 blogs commenting on the NYTimes story this morning, and after clicking through to all 28 I count 13 posts by liberal blogs, 7 by conservatives (with the rest not obviously left or right - your "mileage may vary").   

The conservatives seem to be noting -- as per Reynolds, Sullivan and Kaus -- that the poll also includes less than flattering results for John Kerry and Al Gore.   The favorable ratings for Kerry (26%) and Gore (2819%) are lower than for Bush (29%), although Bush's "not favorable" rating is far higher (55%) than for either Kerry (38%) or Gore (39%27%). 

• BullDogPundit takes his usual look at the demographics of the CBS/NYTimes sample of adults, finds it different from the exit poll of voters in 2004 and comes to his usual conclusion that this discrepancy render this "another crap poll."  MP has always found this line of argument wildly unpersuasive. 

• The ratio of self-identified Democrats (37%) to Republicans (25%) on this survey is bit higher than on the other polls done by CBS and the Times earlier this year (I get an average of 34% Democrat, 29% Republican).  However, the culprit behind the president's declining overall approval rating appears to be his support among Republicans - not the mix of Republicans and Democrats in the survey.  As Franklin notes, the percentage of Republicans that approve of Bush's job rating is a point higher on the CBS/NYT poll (69%) than on the USAToday Gallup poll (68%).  Another way of saying the same thing:  Readjust the party percentages to match the 34%-29% party ID average on recent CBS/NYT polls, and the Bush job approval would rises only slightly (from 31% to 33%)

One reason to avoid this sort of "dynamic weighting" is that CBS/NYT, like most public polls, asks party identification last.  In this case, that means the party identification question came at the end of a very long survey that went into great depth on issues (including gas prices, immigration, the war in Iraq) on which voters expressed considerable unhappiness with the president's performance.  Given the evidence that party ID can change during the course of an interview, it is entirely possible that the apparent skew on this survey was a function of questions that came earlier, not an "oversampling" of Democrats relative to Republicans.

Good News, Bad News & The Blogosphere

The last few days have been busy in terms of new national polls released, yet the commentary on those polls in the blogosphere strikes me as typical.  Partisans on both sides (though mostly on the liberal side this time) seized at the several different polls that appear to tell different stories, when the reality -- as usual -- appears to fall somewhere in the middle.

The gist is that a new survey released yesterday by USAToday/Gallup (story, Gallup report) featured yet another "new low" of 31% for the Bush job approval rating (down from 34% on their survey a week earlier).  Meanwhile, the second survey released by the new CNN/ORC partnership (story, data) showed a slight increase in the Bush job rating, from 32% on a survey conducted two week ago to 34% now.   Of course, neither of these changes were large enough to be statistically significant, although both the Gallup and ORC surveys were conducted on the same dates (May 5-7).   

Meanwhile, yet another survey released by Fox News late last week -- this one of registered voters rather than adults (story, results) -- showed a Bush's job approval rating "rebound[ing] slightly" to 38% from a "record low" of 33% two weeks ago.  This change does appear statistically significant, although the job rating on the previous poll (33%) represented a big drop from surveys conducted by Fox in early April (36%) and mid March (39%).

What caught MP's eye was the way the blogosphere reacted to the 31% result from Gallup.   Not surprisingly, according to USAToday "On Deadline" feature, the Gallup 31% result was "the most-discussed news among bloggers" yesterday afternoon.  And a quick check of that Technorati listing they cited shows commentary on most of the well known liberal blogs and news sites:   TalkingPointsMemo, DailyKos, Atrios, MyDD, Huffington Post, TheLeftCoaster and DemocraticUnderground, among others.   

Not surprisingly, the conservative blogs had little if anything to say about the latest Gallup poll. A quick check shows nothing, for example on Instapundit, Michelle Malkin, Powerline, Captain's Quarters, Little Green Footballs (although I make no claim of having done a truly exhaustive search).   Polipundit -- a site that often features scathing criticism of mainstream media polling -- was the one exception I found. 

On the other hand, I was a bit surprised at how little attention the "rebound" numbers in the Fox poll received from conservative web sites.  Technorati shows very few posts that linked to either the Fox story or the results on any blogs, left or right.  Two prominent conservative sites -- Captain's Quarters and Polipundit -- linked to the Fox poll, but so did the liberal site MyDD.
 

Nonetheless, the pattern is not uncommon.  Partisan political blogs seem to shower attention on polls that show the most dramatic "good news" for their side, and tend to ignore polls showing possibly contradictory results.  The relatively sparse attention paid to the Fox News survey is probably evidence of a general wariness of polls among conservative blogs a period where all seem to bring bad news for the President (a point made vehemently, if indirectly, in this post by Polipundit's DJ Drummond). 

Yet as is also typical, the reality of the change in the Bush job approval rating over the last week or so is neither as dramatically dire as the Gallup result suggests nor as hopeful as the apparent "rebound" in the Fox poll.  Check the latest update to Professor Franklin's graphic (reproduced below) and his commentary yesterday (here, here and here).  Gallup's 31% is on the low side of the usual cloud of variation, Fox's 38% is on the high side.  The long steady decline in the Bush rating continues.  Collectively, the polls released over the last few days leave the rate of descent in Franklin's trend line looking more or less as it did two weeks ago. 

More on Rasmussen, Immigration & Third Parties

Today, thanks to  pollster Scott Rasmussen, we have an update on that hypothetical third-party/immigration question he asked a few weeks ago on one of his automated surveys.  Largely the result of a dialogue on that question involving Mickey Kaus and yours truly, Rasmussen today released new results from a similar hypothetical third-party question, this one involving a third party candidate "promising government-backed universal health care."   This latest result shows that support for such a candidate is nearly as high (28%) as support for a hypothetical third party candidate calling for an "enforcement policy" on immigration (31%).  However, the individuals supporting the respective imaginary candidates were very different.  According to Rasmussen:

While the immigration candidate drew equally from both parties, the Universal Health Care candidate cost the Democratic candidate 18 percentage points while the Republican lost just six.

Thus, Rasmussen presents us with bit more empirical evidence that

Immigration cuts across the typical partisan and ideological lines, [and thus] may have more potential to shake up political status quo than other issues.

So let's give Scott Rasmussen credit for "showing responsiveness to Web commentary rare in a pollster," as Kaus puts it (along with his own thoughts on the meaning of the latest results). 

Let's also point out that the disagreement here between MP, Kaus and Rasmussen is less than meets the eye.   I intended my last post largely to point out that similarly large numbers of Americans have expressed enthusiasm for the notion of a third party without a specific issue attached.  So the level of support for the hypothetical candidates in Rasmussen's questions may be a bit generic, although MP certainly concedes that attitudes on immigration tend to create more division within the party coalitions than between them (as noted in previous posts, the Pew Typology study has remarkably rich data on this point).  MP also agrees with Rasmussen that that the support for third parties on his two questions "probably reflects unhappiness with both parties on particular issues rather than a true opportunity for a third party." 

MP will also concede that in one respect, the immigration issue bears some resemblance to issues that have historically led to the formation of third parties and sometimes even led to party realignment.  Now, I make no pretense of expertise on the history of third party formation and realignments (and I invite those who teach and study it to chime in here with their comments).  However, I once wrote an undergraduate thesis on party realignment, and this discussion reminds me of the theoretical model that Political Scientist James L. Sundquist wrote about in 1973: 

A party system that divides people into two contending political groups on the basis of their attitudes and beliefs about one set of public issues is disturbed by a new issue (or cluster of related issues).  The new issue cleaves the electorate on a different line and hence divides each of the parties internally . . . Either at the outset or as it gathers momentum, the new issue comes to be of such paramount political concern to some proportion of the voters that if it encounters resistance from the parties with which they are affiliated, it overrides all considerations that form the basis of their attachment to those parties.  (From Sundquist, Dynamics of the Party System, pp. 35).

Sundquist theorized that under these conditions, the commitment of the "single issue groups" within each party will override the "centrists" who want to preserve party unity, thus leading to formation of new parties and, possibly, to party realignment. 

I am convinced that immigration policy currently divides both Democrats and Republicans.  I am not convinced that immigration has yet become an issue of as "paramount political concern" as the issues Sundquist wrote about.   

My caution here is that Rasmussen's hypothetical question format has the practical effect of giving such paramount importance to immigration (or universal health care) whether voters feel that way or not.  The structure of the question asks, in effect, what if immigration policy (or some other issue) were the only issue involved in your decision?  Even then, only the imaginary third party candidate gets a clearly identified position (see the comment by "NAR" on my original post).  That's a pretty artificial test. 

Now, MP does not object to such artificial tests.  Campaign pollsters do them all the time.  The trick is to keep the results in perspective.   The more hypothetical and artificial the test, the less likely it is to predict a real outcome. 

It would be a bit less aftificial, for example to test a hypothetical match-up with actual names attached and the positions of all three candidates described.  For example, why not ask about a hypothetical race for president between Hillary Clinton the Democrat, John McCain the Republican and Pat Buchanan running as a third party candidate.  Then read their respective positions on immigration and ask the question again.  Yes, still a bit artificial, but possibly more enlightening.