« When is a Poll Really a Poll? | Main | Iraq the Vote: Hotline/Westhill Takes Up the Challenge »

July 11, 2005

Internet Polling: Unfulfilled Promise

An update on the topic of Internet based political polling: Last week, I noted that some influential academic research on public opinion and the Iraq war had been conducted on the Internet.  It involved  a "panel" of respondents maintained by a company known as Knowledge Networks and recruited using conventional random digit dial (RDD) telephone sampling.  The company offers free Internet access to willing participants that not already online.

However, Knowledge Network's use of "probability sampling" is unique to Internet based surveys.  Many other research companies now offer to poll panels of self-selected respondents that volunteered to participate in their surveys, usually for some sort of monetary incentive.  These non-probability panels are all the rage in commercial market research, but for now, have seen little application in political polling. Why?

In a guest column that appeared in National Journal's Hotline two weeks ago, Doug Usher, a pollster with the Democratic firm the Mellman Group, offered some sensible analysis:

When it comes to quantitative data for political applications -- statistically reliable answers that are projectable onto the entire population -- telephone surveys remain critical. And for good reason: when it comes to cost, population coverage, and predictive accuracy for nearly all political polling, telephone surveys remain the gold standard...

Telephone surveying today -- as always -- has significant problems, and critics are correct for pointing them out. Moreover, the Web may be a critical tool for supplementing telephone polling - indeed, the time may be ripe for the transition. But there are very real impediments to its use in polling in most campaign contexts. Until we overcome those impediments, telephone surveying will continue to dominate the political polling landscape.

Usher goes into depth on three "oft repeated falsehoods about the tradeoffs between Internet and telephone polling."  These are worth reading in full but -- until today -- were trapped behind The Hotline's subscription wall.  However, both the author and the folks at the National Journal have kindly consented to allow MP to reproduce Usher's column in full.  It appears after the jump.  Read it all.

As it happens, The Hotline today also unveiled a web version of its very cool "Blogometer" feature that is now free to non-subscribers.  It's definitely worth a click.   

The following originally appeared in The Hotline:

POLLING ANALYSIS
The Internet's Unfulfilled Promise For Political Polling

By Doug Usher, Hotline guest contributor
Thursday, June 30, 2005

The Internet has dramatically changed the face of most aspects of today's political campaigns. Key elements of campaigns, including fundraising, grassroots network development, and GOTV, now rely on Web innovations for a large and growing portion of the work they do.

At the same time, the Internet has not lived up to its early promise for one key part of campaign consulting: polling. 

And this may not change in the future. Unless major changes are made in e-mail registries, Internet privacy laws, and the way we interact with the Internet, it may never be the same tool that the telephone has been quantitative research for over 50 years.

At the Mellman Group, we've been harnessing the power of the Web in many of our qualitative research applications. Web-based ad testing allows us to very quickly gain input from hundreds of voters in a matter of days, giving us (and campaigns) new flexibility to test ads on the fly.

Online qualitative tracking allows pollsters to advise campaigns about changes on the ground, from all over candidates' home turf. This direct give-and-take helps clients learn about surprise advertising and direct mail, in addition to other changes on the ground that can have an impact on public opinion.

But when it comes to quantitative data for political applications -- statistically reliable answers that are projectable onto the entire population -- telephone surveys remain critical. And for good reason: when it comes to cost, population coverage, and predictive accuracy for nearly all political polling, telephone surveys remain the gold standard.

The argument for moving to the Internet for quantitative survey research usually comes down to a cataloguing of the reasons why it is harder to reach people over the phone: caller ID, growing reluctance to answer the phone, younger people don't have landlines, etc.

But this begs the question: is the Web an adequate replacement for telephone surveys? Here are three oft-repeated falsehoods about the trade-off between Internet and telephone polling.

 

MYTH #1: Telephone polling is no longer accurate, because of low response rates, cell phones, caller ID, and other factors.

It is true that it is becoming harder to reach people via the telephone. Response rates are down (although the exact meaning of "response rates" often lies in the eyes of the beholder). Some voters only have cell phones, which pollsters are not allowed to call. And yes, some people will never take a phone call. All of those are problems, for sure. And they're getting worse.

There was a lot of huffing and puffing about the hard-to-reach cell-phone wielding Deaniacs in Iowa. Around election time, newspapers spilled a lot of ink (and blogs a lot of bandwidth) on the problems that this lack of reachability was going to have.

Despite all of these pre-election complaints, did polling mislead us last election cycle? Were there any results in the last election that deviated in any substantial way from pre-election polling? Naturally, we Democrats would have liked to see Kerry win Ohio and win the general election. But the results were all within the margin of error of public polling (almost exclusively telephone surveys). In the Senate races, Republicans probably hoped that Pete Coors could beat Senator Salazar, and Democrats hoped that Dan Mongiardo could carry out his improbable effort to unseat Jim Bunning. But a review of the pre-election polling shows the actual outcomes to be, frankly, unsurprising.

The nice thing about polling ballot items (as opposed to issue-based polling and corporate market surveys) is that we have a check. Not only was our internal surveying accurate, but the public polls (with just a few exceptions) were as accurate as they have ever been this cycle.

MYTH #2: The high rates of Internet usage mean that Americans are now reachable via the Web.

It is certainly true that more people are using the Web than ever -- in every demographic category. According to the Pew Center on the Internet and American Life, 70 million voting aged Americans go online on an average day, up from 51 million in 2000. Overall 136 million people use the Web -- 67 percent of those over 18.

But many make a fundamental error in interpreting this statistic: just because somebody uses the Web does not mean that they are reachable via the Web. And unless the architecture of the Web is changed dramatically, voters may never be adequately reachable via the Web for useful quantitative research for political campaigns.

All telephone numbers are knowable. Even unlisted numbers can be found, via random digit dial technology. More important, because of area code and exchange standards, phone numbers are knowable within relevant political geography -- including states, counties, and most Congressional Districts. Skilled phone match vendors also do an excellent job getting phone numbers for voter files, which allows pollsters to reach voters at the more local levels of political geography.

E-mail addresses are not universally knowable -- because of the structure of the Internet and e-mail, there are infinite permutations of addresses and domain names. Spammers are continuously trying to circumvent this by generating random e-mail addresses and sending bulk e-mails. While that may be a good way to sell cheap prescription drugs to a small segment of the population, it is much less effective in reaching a representative cross-section of an electorate.

Even with ever-increasing online activity, this problem may never be solvable. Assuming you are able to get through to people, how would you know that they live in the district of interest? There is no connection between e-mail addresses and home addresses, rendering them unreliable in most applications for political polling.

At this point, the only truly "reachable" online population is those who have opted in to survey research. Some have argued that these samples are adequately representative of the country, and even for some large states, when weighted to demographic characteristics. Assuming that this is true, that still leaves nearly all relevant political geography uncovered - most states, and nearly every Congressional District, not to mention every state legislative seat, county executive seat, and mayoralty. And how about finding likely primary voters who are actually properly registered?

Opt-in surveys may be adequate at some point in the future; indeed, they may be adequate now for looking at a very limited political geography. However, they are not now usable for most political polling. And as we move forward, it does not seem particularly likely that their usability will expand to the bulk of political geography in the United States.

MYTH #3: The Internet now allows us to reach voters whom we can't reach by phone.

This is one of the most promising aspects of Internet survey research. If we could supplement telephone survey with Internet research that covers those who are unreachable, we might be able to develop a better research instrument. But the jury is still out on whether it can serve this purpose.

It seems reasonable to assume that "cell-phone only" twenty-somethings spend time online. Additionally, many of those who use technology (like caller ID) to avoid phone calls from people they don't know also have some connection to the Internet.

But - as with the voting population more generally -- the fact that they are online does not make them reachable online. Indeed, it seems unlikely that a person who uses technology to guard their time from telephone surveys will then turn around and spend their time online participating in the very same activity. Do some take surveys online? Probably. But are they representative of the larger population? We just don't know.

For the only population that is truly unreachable by traditional phone methods - those without a landline - Internet polling may provide some inroads. But fundamental questions will remain, as discussed above: are you sure that the people you are speaking with are a) without a landline, b) from the appropriate political geography, and c) are available to you in an adequately representative way? If you can be reasonably sure that those three questions are answered, then Internet polling may provide a nice supplement.

What's the future for political public opinion research over the Web?

The Web is a growing force in qualitative public opinion research in politics -- an invaluable tool for testing ads and exploring the nature of individual public opinion. Our Web-based research is now part of many of the campaigns we work with, allowing them to do research in time frames that were never before possible. In time, more applications will be available online, allowing us to explore different populations quicker than we've ever been able to in the past.

And there are grave concerns about the future of telephone polling, as response rates continue to decline, and reachability narrows.

Yet ten years after Netscape's IPO, the Internet has not lived up to its billing as the panacea for quantitative research. Indeed, I have yet to hear of a single candidate facing a competitive race that used Internet polling as the basis for strategic decisionmaking.

As of now, there are a number of ways for certain types of quantitatively precise surveys to be conducted over the Web. Many organizations, including universities and corporations -- use the Internet to conduct surveys of their employees, students or members. This makes sense, as those organizations tend to have comprehensive and accurate e-mail lists, and are recognized by the membership as credible invitees.

Additionally, at least one national panel has been formed which recruits survey respondents and provides them with access to the Internet in exchange for survey participation. This helps overcome some of the obstacles discussed above, but is also expensive and time consuming. Moreover, its accuracy for quantitative research is largely limited to national efforts.

Telephone surveying today -- as always -- has significant problems, and critics are correct for pointing them out. Moreover, the Web may be a critical tool for supplementing telephone polling - indeed, the time may be ripe for the transition. But there are very real impediments to its use in polling in most campaign contexts. Until we overcome those impediments, telephone surveying will continue to dominate the political polling landscape.

© 2005 by National Journal Group Inc., 600 New Hampshire Avenue, NW, Washington DC 20037.  Any reproduction or retransmission, in whole or in part, is a violation of federal law and is strictly prohibited without the consent of National Journal. All rights reserved.

Related Entries - Sampling Issues

Posted by Mark Blumenthal on July 11, 2005 at 06:14 PM in Sampling Issues | Permalink

Comments

During the 2004 presidential campaign, one pollster, John Zogby, regularly provided Internet-based polls (known as "Zogby Interactive") of key states. I don't know the exact procedures of what he did, but I believe he may have obtained large numbers of volunteer respondents and then weighted his samples on demographics and party ID.

As one type of quality check on Zogby Interactive, I compared the findings of its polls to those of traditional, telephone polls conducted in the same states at around the same time.

As seen in a write-up I did (link below), I think Zogby Interactive acquitted itself pretty well. When the page comes up, you'll see the introduction to my analysis. Be sure to scroll down to see the state-by-state breakdowns.

http://dailykos.com/story/2004/8/4/14813/98242

Posted by: Alan R. | Jul 12, 2005 12:04:24 AM

The comments to this entry are closed.