October 15, 2004
Arianna Huffington, a frequent critic of political polling, is weighing in once again with an indictment of the "downright dangerous impact that polls are having on our democracy," including an assertion about mobile phones that has gotten a lot of attention lately (here, here and here):
Pollsters never call cell phones - of which there are now close to 170 million. And even though most cell phone users also have a hard line, a growing number don't - especially young people, an underpolled and hard-to-gauge demographic that could easily turn out to be the margin of difference in this year's race.
Is she right?
National surveys draw samples of randomly generated telephone numbers, a technique with the power to reach any working wired telephone (or "landline") in the United States. Ariana is right that this method excludes mobile phone numbers* and, obviously, cannot reach those who lack telephone service altogether. If the pool of excluded respondents is large and different from those covered by the sample, the results of the survey could suffer from what survey methodologists call "coverage error." Are polls suffering from coverage error this year? That is a more difficult question.
[*Why do pollsters exclude mobile phone numbers from random digit dial samples? See my note in the comments section]
For the last 20 or so years, telephone surveys have excluded the roughly 5% of U.S. households without any form of home phone service. Those who lack phone service are disproportionately younger, non-white, and lower income, but their numbers are small, they vote at much lower rates than other adults, and pollsters typically weight by age, race and income, so the impact on political polling has been negligible.
However, the willingness of some to disconnect their landlines in favor of mobile phones threatens to increase the size of those missed by telephone samples significantly. So let's try to answer three questions:
1) What percentage of households are wireless only?
2) How are wireless households different?
3) Given the answers to the first two questions, how likely are these differences to affect political polls this year?
What percentage of households are wireless only?
The best data on this issue come from the enormous, high response rate surveys conducted in-person (rather than over the telephone) by government agencies like the Census, the Bureau of Labor Statistics (BLS) and Centers for Disease Control (CDC).
- In the first quarter of 2003, the Consumer Expenditure Survey (BLS) estimated wireless only households at 4.3%, having risen steadily from 0.8% in just two years (n=5,000-8,000 per quarter).
- In the first nine months of 2003, the CDC's National Health Interview Survey (n=23,372) showed 3.6% of U.S. Civilian Households had wireless service but no landline. They also estimated that 3.0% of all adults lived in such households.
- In February of 2004, the Current Population Study (CPS) conducted by BLS and the Census put the number of wireless only households at 6.0% (n=34,219).
How different are wireless only adults?
Wireless-only adults are certainly younger than other adults. The CDC study reported that 6.8% of 18-24 year olds live in wireless only households, compared to much smaller share of those aged 45-64 (1.6%) or 65+ (0.5%).
But beware of concluding that all of the missing respondents are college age. In fact, less than a third (29%) of wireless-only adults are 18-24 - most are either age 25-44 (52%) and many are 45 or older (19%).
Still, if not exclusively college age, wireless only adults are predominantly under age 45 (81%). They also tend to live in large metropolitan areas (82%), earn less than $40,000 annually (66%) and rent rather than own a home (62%; the comparable percentages for adults with a landline are 51% age 18-44, 73% metro area, 39% <$40K and 24% renter).
We do not have data on the political attitudes of wireless only adults, but their demographic profile suggests a Democratic skew. In the latest AP-IPSOS survey, for example, John Kerry leads President Bush by a wide margin among renters (68% to 29%) and those with incomes under $40,000 (60% to 37%). AP-IPSOS had the race even among younger voters, but Kerry is ahead among 18-30 year olds in the recent surveys by CBS (56% to 35%) and the Washington Post (50% to 43% on Oct. 11-13)
We also lack data on the likely turnout of wireless only adults, although their demographic profile suggests they have been much less likely to vote in past elections. According the Current Population Study by the U.S. Census, turnout in the 2000 election was much lower among adults earning less than $35,000 (51%) than those earning over $35,000 (70%), lower among 18-24 year old (36%) than among those over 35 (66%) and lower among renters (44%) than home owners (65%).
How likely are these differences to cause error in the political polls?
We could calculate the "coverage error" that results from excluding wireless-only adults from political polls if we knew two things: (1) How the vote preferences of wireless only adults differ from those with working landlines and (2) the percentage of all likely voters with only wireless service. Unfortunately, both numbers are unknown.
Still, assume for the sake of argument that wireless adults are 5% of the electorate, that a survey of wired households shows a 48%-48% tie and that the missing wireless-only voters prefer John Kerry by a 20-point margin (58% to 38% - a pure but plausible guess based on the numbers for renters, low income, etc). If we were able to include the wireless only adults, it would change the overall preference by only one point - Kerry would lead 48.5% to 47.5%.
Keep in mind that two factors will work to reduce this small potential error: Wireless-only voters are likely to turn out at a lesser rate than those with wired phones, and pollsters typically weight to make up for overall differences in gender, age, race and education.
Of course, that's this year. Things could be very different next time. A recent study by the market research firm In-Stat/MDR estimates wireless only households growing to 30% in 2008. If that estimate holds, telephone polls will face enormous challenges in the very near future.
(Offline sources on the jump)
[Continue with More on Mobile Phones]
Julian V. Luke, Stephen J. Blumberg, and Marcie L. Cynamon. "The Prevalence of Wireless Substitution." Paper presented at the Annual Meeting of the American Association for Public Opinion Research, May 15, 2004, in Phoenix Arizona
Clyde Tucker, Brian Meekins and J. Michael Brick. "Household Telephone Service and Usage Patterns in the United States in 2004." Paper presented at the Annual Meeting of the American Association for Public Opinion Research, May 15, 2004, in Phoenix Arizona
See also the discussion of cell phones by the ABC News Polling Unit
I don't know that "wireless only" is necessarily the only concern that is relevant. It's fairly common among techies - particularly the unmarried ones - to have a land-based phone line exclusively for the purpose of DSL, and to be unreachable via that line because there isn't actually a phone hooked up to it. (I suspect that the number of people for whom this is true is relatively small and impossible to measure. :))
Posted by: aphrael | Oct 15, 2004 1:47:39 PM
"National surveys draw samples of randomly generated telephone numbers .... [T]his method excludes mobile phone numbers.... "
I don't understand. All areas codes - including those used by cell phones - must be known to pollsters. If the numbers generated are truly random, why would this technique miss mobile phones?
Am I missing something?
Posted by: Mithras | Oct 15, 2004 2:27:58 PM
What about people who have both home landlines and cell phones but rarely answer the landline? I know many people who screen their home phone using either an answering machine or caller-ID, but who always pick up their cell phone. These people have come to expect friends and people they know to call the cell phone, and strangers and telemarketers to call the landline. This is pretty standard behavior. I think call screening is as big an issue as cell phones. The two issues together make me uneasy about the accuracy of the polls.
Posted by: Haim Goldman | Oct 15, 2004 2:43:52 PM
Mithras asked a good question:
"All areas codes - including those used by cell phones - must be known to pollsters. If the numbers generated are truly random, why would this technique miss mobile phones?"
In most states, mobile phones use unique exchanges (the three numbers following the area code) that are known (or at least knowable) to pollsters. Yet most pollsters intentiaonlly omit the mobile phone exchanges.
Why? The most important reasons are:
1) FCC's Telephone Consumer Protection Act (TCPA)makes it illegal to make any form of unsolicited call to a mobile phone using an "automated dialer" (which legally includes the equivalent of a speed dial). Almost all survey organizations now use some form of automated dialer to dial respondents.
2) Some states make it illegal to make an unsolicted call when the receipient has to pay for the call.
3) Not surprisingly, because most people pay for incoming calls, response rates on mobile phones are abysmally low.
4) The geographic distinctiveness of area codes blurs quickly for mobile phones, especially in metropolitan "multi-state" areas like New York City and Washington DC. This would matter much more for a statewide survey than a national survey.
Posted by: Mark Blumenthal | Oct 15, 2004 3:46:53 PM
great informative site. thanks for the good work...
question: what's up with rasmussen? are they accurate? i've heard differing opinons on this matter.
Posted by: chris | Oct 15, 2004 4:32:13 PM
Thanks for the response, Mark.
Although it's beyond the scope of the post, I want to echo Haim's comments. What's probably more important than "wireless only" households are "primarily wireless" households, as well as the differences between people who answer their phones when they don't know who is calling and those who let it go to voicemail. I find it plausible to think that people who answer the phone have different political views from those who screen their calls.
Posted by: Mithras | Oct 15, 2004 5:25:38 PM
Thanks for an excellent explanation. Of course, the weighting for age and income groups is probably the most important reason why this factor will not be important in this year's polling accuracy. I really appreciate your evenhandedness in analyzing polling techniques -- good information for both sides.
Posted by: Don | Oct 15, 2004 7:38:41 PM
Thanks for addressing this question. I think however I spot a flaw in your premise (1), which to be fair is a flaw in the way that the question is framed by just about everyone.
5% of people have cut their land lines. Alright. But I would bet another 5% of people *have* a land line, but *never* answer it (they use it just for computer connection or for making certain outgoing calls e.g. overseas), and then another 10%-plus on top of that *always* screen out landline calls in a new way, made possible just in the past few years by the ubiquity of mobile phones, that escapes the possibility of statistical models to filter out. (I know that pollsters have experience with screening; what I'm describing is a whole new level of screening that's become common practice just recently.)
So in this view the situation is much closer now to the tipping point forecast for 2008 by the group you mention.
Obviously everyone knows that pollsters norm for age, gender, income, etc. ... but when you lose 20% of your sample base over a short period of time (5-10 years since mobile phones became prevalent), it is just not possible to know what kind of correction to make.
Sure, renters (for instance) may go for Kerry--but do we really know by how much through any way other than land-line telephone polls? If not, we're feeding bad data back into our predictive loop ...
Posted by: Ottoe | Oct 15, 2004 10:39:06 PM
Great, just read over the other comments, good to see that great minds think alike.
Another problem with the above (again offered in what John Kerry would call a "constructive" spirit):
In the first nine months of 2003, the CDC's National Health Interview Survey (n=23,372) showed 3.6% of U.S. Civilian Households had wireless service but no landline. They also estimated that 3.0% of all adults lived in such households.
In February of 2004, the Current Population Study (CPS) conducted by BLS and the Census put the number of wireless only households at 6.0% (n=34,219).
5/2003 = 3.6%
2/2004 = 6.0%
10/2004 = 8.1 %
And that's assuming linear growth. Of course the growth of such households is more likely exponential. Looks like the baseline figure of wireless households we may be dealing with come election time may be near twice your estimate.
If you doubt such a boom, remember all the hype/public discussion of phone number portability, etc: that was really the moment when cutting the land line became a possiblity known to the masses (hence the jumping-off point for a fad).
Posted by: Ottoe | Oct 15, 2004 10:49:41 PM
Haim Goldman's Comment is the big point that I think you're missing. I'll bet that the number of landline owners who rarely or never answer their landline is at least equal to the number who don't own a landline. This demographic conducts its business and social relations via cell; the landline is there primarily for outgoing calls at home. The landline may even be disused for most outgoing calls, given the financial incentive in cell phone plans to use free minutes, and retained only for emergencies (or when the cell battery goes dead and you don't have your charger around) or by convention. When you don't expect incoming landline calls, you are far more likely to screen them.
Another point you undervalue - turnout among young people. While I am not arguing that it will be equivalent to turnout among all adults, surely you're aware of extensive anecdotal evidence of young people claiming that they are more attuned to politics and the importance of voting in the post-9/11 world?
Posted by: gabbneb | Oct 16, 2004 9:55:15 AM
What some of the comments miss is that the more voters adopt mobile phones, the more their political responses will regress toward the mean.
The early adopters may be young urban renters, but nowadays you see Mexican gardeners and little old ladies with cell phones. As they spread, won't the mobile phone user profile tend to resemble the population as a whole?
I'll hypothesize that the number of people with land lines who screed calls or won't talk to pollsters is a bigger problem than the "mobile only" types.
This picture is rapidly changing, and it can't be solved my another telephone survey. It's also possible that voter participation will be way up this year so all "likely voter" models based on past behavior are suspect. Except for Karl Rove's 4 million evangelicals, traditional wisdom says this helps the Dems.
This is going to be a cliffhanger.
Posted by: Grumpy Old Man | Oct 16, 2004 10:39:48 AM
Here's a list of possible error factors in political polling this year:
Unreached mobile phone users.
People with caller ID who screen their calls.
People who won't talk to pollsters -- a growing number, I'll bet.
Unlikely voters who may unexpectedly show up this year, including Karl Rove's 4 million evangelicals and verybody else who votes because of the great interest of this watershed campaign.
Possibly depressed turnout among blacks because Kerry hasn't connected with them or because of gay marriage, and among a rainbow of Floridians who are who knows where because of the hurricanes.
Posted by: Grumpy Old Man | Oct 16, 2004 11:04:15 AM
I'm wondering if pollsters can get through to people who are on the Do Not Call Lists? There are a lot of people on those lists. Would this effect the poll?
Posted by: cw | Oct 16, 2004 11:31:37 AM
"Keep in mind that two factors will work to reduce this small potential error: Wireless-only voters are likely to turn out at a lesser rate than those with wired phones, and pollsters typically weight to make up for overall differences in gender, age, race and education."
Not this year, bubba. The greater the turnout, the better for Democrats and this year the turnout among college students will be huge...and hugely anti-Bush. The weighting process that you cite will probably undercount them because of the conventional wisdom that they are less likely to vote. Kerry is going to win by a surprisingly large margin.
Posted by: bendan | Oct 16, 2004 12:04:44 PM
In your conclusion -- caveats omitted:
"If we were able to include the wireless only adults, it would change the overall preference by only one point - Kerry would lead 48.5% to 47.5%."
...only one point, given how close things are, sounds like a lot to me.
Posted by: DonBoy | Oct 16, 2004 1:01:47 PM
Survey takers can call numbers listed on the do not call list. But, some people who are on these lists assume that survey takers are on the same level as the marketers they are avoiding, under the law. So there may be some on the list avoiding the calls from the pollsters. A better way to show that college age voters are under represented is te fact that random dialers rarely pick up on dorms on some campuses. Some of my quietest 5-8pm's have been when I was living in the dorms of a private college. This would also be a factor at small colleges that can have thier phone lines set up like a small buisiness system with it's own three digit exchange and the rooms acting as extensions (in a way) off of the main system.
Posted by: Thom | Oct 16, 2004 1:02:22 PM
You mention something important early on, but then never quantify it: Mobile phone users are more likely to be in urban areas.
I'm surprised that population density isn't one of the variables used to adjust poll results. Isn't population density one of the strongest correlatives with Democratic leaning?
If so, mobile phone users, whose service is always better in denser areas with more closely-packed transmitters, should lean left, no?
Posted by: JW | Oct 16, 2004 2:18:38 PM
What I find interesting is the high level of unemployment among the surveyed. Perhaps, they are home and bored, whereas working folks don't answer the phone. Even if you adjust these polls for gender, etc., you're still oversampling the idle.
With regard to homes that lack landlines, my choice to go wireless was motivated by the fact that the only people who called me on my landline were telemarketers. My friends and associates always called my cell. I doubt that this is unusual. The comment on people with cells never answering their landline is very appropriate and true in my experience.
Posted by: Rick K | Oct 16, 2004 2:28:51 PM
Anacdotal evidence alert, but im 28, have no land line and have 5 or 6 friends who dont either. The ones without landlines tend to be techy, make more money, own condos/houses, and in fact trend republican/libertarian. I think it might be dangerous to make demographic assumptions about a given behavior like that. By the rationale presented you could prove that Young Republican tend to vote for Kerry.
I'd say someone should conduct a poll, but...
Posted by: Mark Buehner | Oct 16, 2004 2:30:50 PM
"The greater the turnout, the better for Democrats and this year the turnout among college students will be huge...and hugely anti-Bush."
It's not 1968 anymore.
Posted by: Floyd McWilliams | Oct 16, 2004 2:50:35 PM
How do polls take into account military people? Seems to me they likely have a pro-Bush tilt this year. (During the civil war, it was furloughed soldiers voting that might have decisively tipped the balance for Lincoln). But are they taken into account in phone polls if they are overseas or in barracks?
Second, what about non-military U.S. citizens who are out of the country? Do they tend to tilt for one party or the other, and are they taken into account in polls?
Thanks for any insight you can provide,
Posted by: docpro | Oct 16, 2004 4:52:04 PM
1) If the choice to go wireless trends together with renting then it also trends together with non-registration, since renters are less likely to be registered to vote (at least at their current residence). That's at least a reasonable argument that, at current levels of wireless only households, the selection bias due to cell phone use is not very significant.
2) Do Not Call Lists do not apply to polling. Only to commercial solicitation.
Posted by: scgarris | Oct 16, 2004 5:16:47 PM
turn some states blue (then watch the animations)
Posted by: Brian N. | Oct 16, 2004 8:25:06 PM
1. Young people, who usually rent, often go wireless-only. They are more likely to go out when not working or sleeping. They often have roommates: cell phones prevent fights over phone use. They change their residences more frequently than those who have a mortgage, a career, a family, and a couple decades' worth of accumulated possessions to keep them in one place.
2. Cell phones are more convenient for those whose jobs involve much travel on a daily basis (housekeepers, private tutors, etc.). I've no hard evidence, but perhaps such individuals reason that so long as they have a cell phone, they might as well buy a family plan for the others in the household instead of paying for the land line.
3. As Mark Blumenthal mentions, "The geographic distinctiveness of area codes blurs quickly for mobile phones, especially in metropolitan "multi-state" areas like New York City and Washington DC." Two of my friends have Philadelphia area codes, but live in Stamford and New York. One has a Berkshire County (MA) area code but lives in Boston. Again, this is a result of the high mobility of young people.
Posted by: g.b.s. | Oct 16, 2004 10:42:18 PM
Is there a systematic bias of the raw data demographic
information? If cell phones are a problem and if cell
phone-only use skews young, then virtuallly all the
polls should consistently (apart from sampling) have
too few young people. Is that the case? and is this
a significant undersampling?
IF that is the case, then the polls are all off, since
they are excluding a segment of the population. You
can't reweight for that.
Posted by: Matt Newman | Oct 16, 2004 11:32:52 PM
The comments to this entry are closed.