Mystery Pollster

An "Immigration-Enforcement" Third Party?

Yesterday, our friend Mickey Kaus highlighted a question from a recent Rasmussen automated survey worth examining a bit more closely.  The question asks voters to choose between "generic" Republican and Democratic candidates (with no stated immigration position) and a third party candidate that takes a hard-line anti-immigration position.  The third-party candidate gets 30% of the vote, leading both Rasmussen and Kaus to speculate about the potential power of immigration to reshape our politics.  Let me suggest an alternative:  It may simply confirm the desire for a third party (at least in theory) by a large number of Americans regardless of the issues involved.   

Courtesy of Scott Rasmussen, here is the full text of the two questions at issue (and remember that Rasmussen currently weights his survey a few points more Republican than other national samples of adults):

If the 2008 Presidential Election were held today, would you vote for the Republican candidate or the Democratic candidate?        
44% Democrat        
32% Republican        
7% Other        
17% Not sure

Suppose a third party candidate ran in 2008 and promised to build a barrier along the Mexican border and make enforcement of immigration law his top priority.  Would you vote for the Republican, the Democrat, or the third party candidate?
31% Democrat
21% Republican
30% Third party/other
18% Not sure

On his web site, Scott Rasmussen concludes:

This result probably reflects unhappiness with both parties on the immigration issue rather than a true opportunity for a third party. Historically, issues that drive third party candidates get co-opted by one of the major parties as they demonstrate popular appeal

Blogging at RealClearPolitics, he adds that the result "be taken as an indication of the [immigration] issue's power rather than a literal projection of election outcomes." 

Fair enough.  And while there is good evidence elsewhere (especially here) that the immigration issue produces more division within the two political parties than between them, let me suggest another reason to be careful about reading too much into this particular question.  It may tell us as much about the strong general desire for a third party candidate as it does about the power of the immigration issue specifically.   

Consider this result from the just released NBC/Wall Street Journal poll, which shows 45% favor the idea of a "new independent political party" and 29% oppose:

Tell me whether you would strongly favor, mildly favor, feel neutral about, mildly oppose, or strongly oppose this change:  Build a new independent political party to run a credible candidate for president.
31% strongly favor
14% mildly favor
24% feel neutral
12% mildly oppose
17% strongly oppose 
2% not sure

Or consider these questions asked by Gallup survey in 2003:

In your view, do the Republican and Democratic parties do an adequate job of representing the American people, or do they do such a poor job that a third major party is needed? (10/10-12/2003, n=1,004)
56% Do an adequate job
40% Third party needed
4% Don't know/refused

Have you ever voted for an independent or a third party candidate for president, that is, a candidate for president who was not either a Republican or a Democrat? (9/19-21/2003, n=1,003)
28% Yes, have
71% No, have not
1% Don't know/refused

It is also worth expanding on one of Kaus' caveats:  "Candidates with appealing specifics often beat undefined, generic party choices."  That is true, in that questions that inform respondents about the specific issue positions of specific named candidates typically get more of a response (e.g. fewer undecideds) than questions posing only "generic" choices.  However, Rasmussen's question is a bit unusual in that it includes both types of choices on the same question.  To be honest, I have not seen that done before and am not quite sure what to make of the result. 

Again, I do not want to minimize the potential for the immigration issue to divide the bases of both parties, particularly the conservative Republican base.   And Rasmussen reports that his hypothetical tough-on-immigration third party candidate divides self-identified conservatives, getting 35% to the generic Republican's 36%, while liberals still overwhelming prefer the Democrat (65%) to the third party candidate (19%).**  That result is worth pondering, even though, as Rasmussen appropriately warns, we should not consider it "a literal projection of election outcomes."

**Although note that self-identified conservatives outnumber liberals in Rasmussen's sample by roughly two to one (34% to 17%).

An Online Poll on an Online Activity

One of the gratifying things about writing this blog is the collective power of Mystery Pollster readers.  Last week, I emailed some questions to Zogby International about the methodology they used for a recent poll conducted on behalf of the online gambling industry. The poll had been the subject of a column by Carl Bialik for the Wall Street Journal Online, something I discussed in a post last Friday.  Zogby's spokesman ignored my emails.  However, over the weekend MP reader Ken Alper reported in a comment that he had been a respondent to the Zogby gambling poll. He also confirmed my hunch: Zogby conducted the survey online. That fact raises even more questions about the potential for bias in the Zogby results.

Why is it important that the survey was conducted online?

1) This survey is not based on a "scientific" random sample -- The press release posted on the web site of the trade group that paid for the poll makes the claim that it is a "scientific poll" of "likely voters."   As we have discussed here previously, we use the term scientific to describe a poll based on a random probability sample, one in which all members of a population (in this case, all likely voters) have an equal or known chance of being selected at random. 

In this case only individuals that had previously joined the Zogby panel of potential respondents had that opportunity.  As this article on the Zogby's web site explains, their online samples are selected from "a database of individuals who have registered to take part in online polls through solicitations on the company's Web site, as well as other Web sites that span the political spectrum."    In other words, most of the members of the panel saw a banner ad on a web site and volunteered to participate.  You can volunteer too - just use this link.

Zogby claims that "many individuals who have participated in Zogby's telephone surveys also have submitted e-mail addresses so they may take part in online polls."  Such recruitment might help make Zogby's panel a bit more representative, but it certainly does not transorm it into a random sample.  Moreover, he tells us nothing about the percentage of such recruits in his panel or the percentage of telephone respondents that typically submit email addresses.  Despite Zogby's bluster, this claim does not come close to making his "database" a projective random sample of the U.S. population. 

2) The survey falsely claims to have a "margin of error" -- Specifically, the gambling survey press release reports a margin of error  0.6 percentage points.  That happens to be exactly the margin you get when you plug the sample size (n=30,054) into the formula for a confidence interval that assumes "simple random sampling."  In other words, to have a "margin of error" the survey has to be based on a random probability sample.  But see #1.  This is not a random sample. 

Several weeks ago, I wrote about an online panel survey conducted on behalf of the American Medical Association that similarly claimed a "margin of error."  But in that case, the pollster quickly corrected the "inadvertent" error when brought to his attention:

We do not, and never intended to, represent [the AMA spring break survey] it as a probability study and in all of our disclosures very clearly identified it as a study using an online panel.  We reviewed our methodology statement and noticed an inadvertent declaration of sampling error.

I emailed Zogby spokesman Fritz Wenzel last Thursday to ask how they justified the term "scientific" and the claim of a "margin of error."  I have not yet received any response.

3) The survey press release fails to disclose that it was conducted online -- Check the standards of disclosure of the National Council on Public Polls, standards adhered to by most of the major public pollsters.  They specifically require that "all reports of survey findings" include a reference to the "method of obtaining the interviews (in-person, telephone or mail)."  Obviously, Zogby's gambling poll release includes no such reference.

Now, it is certainly possible that the press release in question was authored by the client (the gambling trade group) and not by Zogby International.  My email to Fritz Wenzel included this question.  The subsequent silence of the Zogby organization on this issue is odd since most pollsters, including my own firm, reserve the right (usually by contract) to publicly correct any misrepresentations of data made by our clients. 

4) This online survey concerned the regulation of online activity -- Even surveys conducted using random sampling are subject to other kinds of errors.  Specifically, when those not covered by the sample or those who do not respond to the survey have systematically different opinions than those included in the survey, the results will be biased.  At a minimum, Zogby's methodology can include only Americans that are online.  More important, it does not randomly sample online Americans.  Rather, it samples from a "database" of individuals that opted in, many because they saw a banner advertisement on a web page.  As such, these individuals are almost by definition among the heaviest and most adventurous of online users. 

While MP is intrigued by new methodologies that claim to manipulate the selection or weighting of results from such a sample to resemble the overall population, he warns readers of what should be obvious:  The potential for bias using such a technique will be greatest when the survey topic is some aspect of online behavior or the Internet itself.  With these topics, the differences between the panel and the population of interest are likely to be greatest. 

I searched for but could not find a random sample survey that could show the relationship between attitudes on online gambling and time spent online.  However, I did find a data on potential government restrictions that allow for such analysis in a survey conducted in the summer of 2002 by the Pew Internet & American Life Project.  The survey asked a question about government monitoring of email and also asked respondents how often they went online.  A cross-tabulation shows an unsurprising pattern.  Those who went online "daily" opposed government monitoring of email by a margin of twelve points (42% yes, 54% no).  Those who were offline altogether supported monitoring by ten points (49% yes, 39% no).  Heavy online users are more skeptical of government regulation of the Internet than the population as a whole. 

Now obviously, we can only speculate whether such a pattern might apply to the regulation of Internet gambling, but common sense suggests that it is a strong possibility.  And it provides yet another reason for skepticism about the results of this particular Zogby poll. 

So to sum up what we have learned:

In Carl Bialik's column, American Association for Public Opinion Research (AAPOR)  President Cliff Zukin described the survey questions as "leading and biased."  Further, the survey release failed to disclose that it was conducted online and made the statistically indefensible claim to be a "scientific poll" with a "margin of error."   The failure to disclose the use of sampling from an online panel is particularly deceptive given that an online activity was the focus of the survey.  Add that all up and you get one remarkably misleading poll release. 

This story also presents a tough question for Carl Bialik's editors at the Wall Street Journal Online:   Is it statistically defensible to report a "margin of error" for this non-probability sample?  And if not, why does the Wall Street Journal Online allow Zogby to routinely report a "margin of error" for the Internet panel surveys that the Journal sponsors?

MP on WTWP

Short notice, I know, by yours truly will be a guest on Washington Post Radio this afternoon at about 2:30, along with Post pollsters Richard Morin and Claudia Deane.   Those in the DC area can tune in at 1500 AM or 107.7 FM.  You can listen to the show live via streaming audio at this link.   

CNN & ORC: The Real "Breaking News?"

Thanks to the loyal reader who alerted MP to this just released poll (story, full results) by CNN which puts the Bush job approval rating at yet another new low of only 32%. 

But wait, despite yet another "new low" story (or perhaps because of it), MP readers may be less interested in the results of the survey than organization that conducted it:  The Opinion Research Corporation (ORC).   Remember that CNN recently severed its long-time survey partnership with the Gallup Organization and USAToday.  Could ORC be the new partner and Gallup's replacement?   It sure looks that way.

One who knows tells MP that ORC is one of the original "big four" of American commercial polling firms, along with Gallup, Harris and Roper.   Like Roper, they have focused for most of their history on surveys conducted for corporate clients.  Their web site tells us that ORC is an international  "research and consulting firm," founded in 1938, with clients in "both the public and private sectors."   A Google News search turns up a sampling of some of their recent projects and clients.   

A warning to MP readers:  Be careful of making too much of the "new low" comparisons to other recent surveys, particularly the  most recent surveys conducted by Gallup and the now defunct CNN/USAToday/Gallup partnership.   CNN is still the sponsor, and while the wording may be the same, the pollster and calling centers are different.  As we have discussed previously (especially here and here), different polls can have different house effects that make for slightly but consistently different results.   See especially the posts on this topic by Charles Franklin and Robert Chung. 

On this point, note that in its PDF release, CNN appears to be separating the latest results from the "CNN/USAToday/Gallup Trends."

UPDATE (4/25):  As he notes in the comments, Charles Franklin has posted his own thoughts on the new CNN data collected by ORC as well as yet another update of his job approval graphic.   He also notes that the new CNN pdf release fails to include results for demographic items and makes the following point, with which I totally agree: 

EVERY reputable pollster should be willing to release the topline results for their ENTIRE survey, not just the items they include in their story. It is crucial for credibility and for more informed interpretation of the poll results. (I'm not talking about embargoing results for later stories which is fine-- the demographics don't fall under any embargo and should be released immediately.)

[Typos corrected]