Arianna Huffington, a frequent critic of political polling, is weighing in once again with an indictment of the "downright dangerous impact that polls are having on our democracy," including an assertion about mobile phones that has gotten a lot of attention lately (here, here and here):
Pollsters never call cell phones – of which there are now close to 170 million. And even though most cell phone users also have a hard line, a growing number don’t – especially young people, an underpolled and hard-to-gauge demographic that could easily turn out to be the margin of difference in this year’s race.
Is she right?
National surveys draw samples of randomly generated telephone numbers, a technique with the power to reach any working wired telephone (or "landline") in the United States. Ariana is right that this method excludes mobile phone numbers* and, obviously, cannot reach those who lack telephone service altogether. If the pool of excluded respondents is large and different from those covered by the sample, the results of the survey could suffer from what survey methodologists call "coverage error." Are polls suffering from coverage error this year? That is a more difficult question.
[*Why do pollsters exclude mobile phone numbers from random digit dial samples? See my note in the comments section]
For the last 20 or so years, telephone surveys have excluded the roughly 5% of U.S. households without any form of home phone service. Those who lack phone service are disproportionately younger, non-white, and lower income, but their numbers are small, they vote at much lower rates than other adults, and pollsters typically weight by age, race and income, so the impact on political polling has been negligible.
However, the willingness of some to disconnect their landlines in favor of mobile phones threatens to increase the size of those missed by telephone samples significantly. So let’s try to answer three questions:
1) What percentage of households are wireless only?
2) How are wireless households different?
3) Given the answers to the first two questions, how likely are these differences to affect political polls this year?
What percentage of households are wireless only?
The best data on this issue come from the enormous, high response rate surveys conducted in-person (rather than over the telephone) by government agencies like the Census, the Bureau of Labor Statistics (BLS) and Centers for Disease Control (CDC).
- In the first quarter of 2003, the Consumer Expenditure Survey (BLS) estimated wireless only households at 4.3%, having risen steadily from 0.8% in just two years (n=5,000-8,000 per quarter).
- In the first nine months of 2003, the CDC’s National Health Interview Survey (n=23,372) showed 3.6% of U.S. Civilian Households had wireless service but no landline. They also estimated that 3.0% of all adults lived in such households.
- In February of 2004, the Current Population Study (CPS) conducted by BLS and the Census put the number of wireless only households at 6.0% (n=34,219).
How different are wireless only adults?
Wireless-only adults are certainly younger than other adults. The CDC study reported that 6.8% of 18-24 year olds live in wireless only households, compared to much smaller share of those aged 45-64 (1.6%) or 65+ (0.5%).
But beware of concluding that all of the missing respondents are college age. In fact, less than a third (29%) of wireless-only adults are 18-24 – most are either age 25-44 (52%) and many are 45 or older (19%).
Still, if not exclusively college age, wireless only adults are predominantly under age 45 (81%). They also tend to live in large metropolitan areas (82%), earn less than $40,000 annually (66%) and rent rather than own a home (62%; the comparable percentages for adults with a landline are 51% age 18-44, 73% metro area, 39% <$40K and 24% renter).
We do not have data on the political attitudes of wireless only adults, but their demographic profile suggests a Democratic skew. In the latest AP-IPSOS survey, for example, John Kerry leads President Bush by a wide margin among renters (68% to 29%) and those with incomes under $40,000 (60% to 37%). AP-IPSOS had the race even among younger voters, but Kerry is ahead among 18-30 year olds in the recent surveys by CBS (56% to 35%) and the Washington Post (50% to 43% on Oct. 11-13)
We also lack data on the likely turnout of wireless only adults, although their demographic profile suggests they have been much less likely to vote in past elections. According the Current Population Study by the U.S. Census, turnout in the 2000 election was much lower among adults earning less than $35,000 (51%) than those earning over $35,000 (70%), lower among 18-24 year old (36%) than among those over 35 (66%) and lower among renters (44%) than home owners (65%).
How likely are these differences to cause error in the political polls?
We could calculate the "coverage error" that results from excluding wireless-only adults from political polls if we knew two things: (1) How the vote preferences of wireless only adults differ from those with working landlines and (2) the percentage of all likely voters with only wireless service. Unfortunately, both numbers are unknown.
Still, assume for the sake of argument that wireless adults are 5% of the electorate, that a survey of wired households shows a 48%-48% tie and that the missing wireless-only voters prefer John Kerry by a 20-point margin (58% to 38% – a pure but plausible guess based on the numbers for renters, low income, etc). If we were able to include the wireless only adults, it would change the overall preference by only one point – Kerry would lead 48.5% to 47.5%.
Keep in mind that two factors will work to reduce this small potential error: Wireless-only voters are likely to turn out at a lesser rate than those with wired phones, and pollsters typically weight to make up for overall differences in gender, age, race and education.
Of course, that’s this year. Things could be very different next time. A recent study by the market research firm In-Stat/MDR estimates wireless only households growing to 30% in 2008. If that estimate holds, telephone polls will face enormous challenges in the very near future.
(Offline sources on the jump)
[Continue with More on Mobile Phones]
Offline sources:
Julian V. Luke, Stephen J. Blumberg, and Marcie L. Cynamon. "The Prevalence of Wireless Substitution." Paper presented at the Annual Meeting of the American Association for Public Opinion Research, May 15, 2004, in Phoenix Arizona
Clyde Tucker, Brian Meekins and J. Michael Brick. "Household Telephone Service and Usage Patterns in the United States in 2004." Paper presented at the Annual Meeting of the American Association for Public Opinion Research, May 15, 2004, in Phoenix Arizona
See also the discussion of cell phones by the ABC News Polling Unit
I don’t know that “wireless only” is necessarily the only concern that is relevant. It’s fairly common among techies – particularly the unmarried ones – to have a land-based phone line exclusively for the purpose of DSL, and to be unreachable via that line because there isn’t actually a phone hooked up to it. (I suspect that the number of people for whom this is true is relatively small and impossible to measure. :))
“National surveys draw samples of randomly generated telephone numbers …. [T]his method excludes mobile phone numbers…. ”
I don’t understand. All areas codes – including those used by cell phones – must be known to pollsters. If the numbers generated are truly random, why would this technique miss mobile phones?
Am I missing something?
What about people who have both home landlines and cell phones but rarely answer the landline? I know many people who screen their home phone using either an answering machine or caller-ID, but who always pick up their cell phone. These people have come to expect friends and people they know to call the cell phone, and strangers and telemarketers to call the landline. This is pretty standard behavior. I think call screening is as big an issue as cell phones. The two issues together make me uneasy about the accuracy of the polls.
Mithras asked a good question:
“All areas codes – including those used by cell phones – must be known to pollsters. If the numbers generated are truly random, why would this technique miss mobile phones?”
In most states, mobile phones use unique exchanges (the three numbers following the area code) that are known (or at least knowable) to pollsters. Yet most pollsters intentiaonlly omit the mobile phone exchanges.
Why? The most important reasons are:
1) FCC’s Telephone Consumer Protection Act (TCPA)makes it illegal to make any form of unsolicited call to a mobile phone using an “automated dialer” (which legally includes the equivalent of a speed dial). Almost all survey organizations now use some form of automated dialer to dial respondents.
2) Some states make it illegal to make an unsolicted call when the receipient has to pay for the call.
3) Not surprisingly, because most people pay for incoming calls, response rates on mobile phones are abysmally low.
4) The geographic distinctiveness of area codes blurs quickly for mobile phones, especially in metropolitan “multi-state” areas like New York City and Washington DC. This would matter much more for a statewide survey than a national survey.
great informative site. thanks for the good work…
question: what’s up with rasmussen? are they accurate? i’ve heard differing opinons on this matter.
Thanks for the response, Mark.
Although it’s beyond the scope of the post, I want to echo Haim’s comments. What’s probably more important than “wireless only” households are “primarily wireless” households, as well as the differences between people who answer their phones when they don’t know who is calling and those who let it go to voicemail. I find it plausible to think that people who answer the phone have different political views from those who screen their calls.
Thanks for an excellent explanation. Of course, the weighting for age and income groups is probably the most important reason why this factor will not be important in this year’s polling accuracy. I really appreciate your evenhandedness in analyzing polling techniques — good information for both sides.
Thanks for addressing this question. I think however I spot a flaw in your premise (1), which to be fair is a flaw in the way that the question is framed by just about everyone.
5% of people have cut their land lines. Alright. But I would bet another 5% of people *have* a land line, but *never* answer it (they use it just for computer connection or for making certain outgoing calls e.g. overseas), and then another 10%-plus on top of that *always* screen out landline calls in a new way, made possible just in the past few years by the ubiquity of mobile phones, that escapes the possibility of statistical models to filter out. (I know that pollsters have experience with screening; what I’m describing is a whole new level of screening that’s become common practice just recently.)
So in this view the situation is much closer now to the tipping point forecast for 2008 by the group you mention.
Obviously everyone knows that pollsters norm for age, gender, income, etc. … but when you lose 20% of your sample base over a short period of time (5-10 years since mobile phones became prevalent), it is just not possible to know what kind of correction to make.
Sure, renters (for instance) may go for Kerry–but do we really know by how much through any way other than land-line telephone polls? If not, we’re feeding bad data back into our predictive loop …
Great, just read over the other comments, good to see that great minds think alike.
Another problem with the above (again offered in what John Kerry would call a “constructive” spirit):
You write:
In the first nine months of 2003, the CDC’s National Health Interview Survey (n=23,372) showed 3.6% of U.S. Civilian Households had wireless service but no landline. They also estimated that 3.0% of all adults lived in such households.
In February of 2004, the Current Population Study (CPS) conducted by BLS and the Census put the number of wireless only households at 6.0% (n=34,219).
So:
5/2003 = 3.6%
2/2004 = 6.0%
10/2004 = 8.1 %
And that’s assuming linear growth. Of course the growth of such households is more likely exponential. Looks like the baseline figure of wireless households we may be dealing with come election time may be near twice your estimate.
If you doubt such a boom, remember all the hype/public discussion of phone number portability, etc: that was really the moment when cutting the land line became a possiblity known to the masses (hence the jumping-off point for a fad).
Haim Goldman’s Comment is the big point that I think you’re missing. I’ll bet that the number of landline owners who rarely or never answer their landline is at least equal to the number who don’t own a landline. This demographic conducts its business and social relations via cell; the landline is there primarily for outgoing calls at home. The landline may even be disused for most outgoing calls, given the financial incentive in cell phone plans to use free minutes, and retained only for emergencies (or when the cell battery goes dead and you don’t have your charger around) or by convention. When you don’t expect incoming landline calls, you are far more likely to screen them.
Another point you undervalue – turnout among young people. While I am not arguing that it will be equivalent to turnout among all adults, surely you’re aware of extensive anecdotal evidence of young people claiming that they are more attuned to politics and the importance of voting in the post-9/11 world?
What some of the comments miss is that the more voters adopt mobile phones, the more their political responses will regress toward the mean.
The early adopters may be young urban renters, but nowadays you see Mexican gardeners and little old ladies with cell phones. As they spread, won’t the mobile phone user profile tend to resemble the population as a whole?
I’ll hypothesize that the number of people with land lines who screed calls or won’t talk to pollsters is a bigger problem than the “mobile only” types.
This picture is rapidly changing, and it can’t be solved my another telephone survey. It’s also possible that voter participation will be way up this year so all “likely voter” models based on past behavior are suspect. Except for Karl Rove’s 4 million evangelicals, traditional wisdom says this helps the Dems.
This is going to be a cliffhanger.
Here’s a list of possible error factors in political polling this year:
Unreached mobile phone users.
People with caller ID who screen their calls.
People who won’t talk to pollsters — a growing number, I’ll bet.
Unlikely voters who may unexpectedly show up this year, including Karl Rove’s 4 million evangelicals and verybody else who votes because of the great interest of this watershed campaign.
Possibly depressed turnout among blacks because Kerry hasn’t connected with them or because of gay marriage, and among a rainbow of Floridians who are who knows where because of the hurricanes.
I’m wondering if pollsters can get through to people who are on the Do Not Call Lists? There are a lot of people on those lists. Would this effect the poll?
“Keep in mind that two factors will work to reduce this small potential error: Wireless-only voters are likely to turn out at a lesser rate than those with wired phones, and pollsters typically weight to make up for overall differences in gender, age, race and education.”
Not this year, bubba. The greater the turnout, the better for Democrats and this year the turnout among college students will be huge…and hugely anti-Bush. The weighting process that you cite will probably undercount them because of the conventional wisdom that they are less likely to vote. Kerry is going to win by a surprisingly large margin.
In your conclusion — caveats omitted:
“If we were able to include the wireless only adults, it would change the overall preference by only one point – Kerry would lead 48.5% to 47.5%.”
…only one point, given how close things are, sounds like a lot to me.
Survey takers can call numbers listed on the do not call list. But, some people who are on these lists assume that survey takers are on the same level as the marketers they are avoiding, under the law. So there may be some on the list avoiding the calls from the pollsters. A better way to show that college age voters are under represented is te fact that random dialers rarely pick up on dorms on some campuses. Some of my quietest 5-8pm’s have been when I was living in the dorms of a private college. This would also be a factor at small colleges that can have thier phone lines set up like a small buisiness system with it’s own three digit exchange and the rooms acting as extensions (in a way) off of the main system.
You mention something important early on, but then never quantify it: Mobile phone users are more likely to be in urban areas.
I’m surprised that population density isn’t one of the variables used to adjust poll results. Isn’t population density one of the strongest correlatives with Democratic leaning?
If so, mobile phone users, whose service is always better in denser areas with more closely-packed transmitters, should lean left, no?
What I find interesting is the high level of unemployment among the surveyed. Perhaps, they are home and bored, whereas working folks don’t answer the phone. Even if you adjust these polls for gender, etc., you’re still oversampling the idle.
With regard to homes that lack landlines, my choice to go wireless was motivated by the fact that the only people who called me on my landline were telemarketers. My friends and associates always called my cell. I doubt that this is unusual. The comment on people with cells never answering their landline is very appropriate and true in my experience.
Anacdotal evidence alert, but im 28, have no land line and have 5 or 6 friends who dont either. The ones without landlines tend to be techy, make more money, own condos/houses, and in fact trend republican/libertarian. I think it might be dangerous to make demographic assumptions about a given behavior like that. By the rationale presented you could prove that Young Republican tend to vote for Kerry.
I’d say someone should conduct a poll, but…
“The greater the turnout, the better for Democrats and this year the turnout among college students will be huge…and hugely anti-Bush.”
It’s not 1968 anymore.
Hi,
How do polls take into account military people? Seems to me they likely have a pro-Bush tilt this year. (During the civil war, it was furloughed soldiers voting that might have decisively tipped the balance for Lincoln). But are they taken into account in phone polls if they are overseas or in barracks?
Second, what about non-military U.S. citizens who are out of the country? Do they tend to tilt for one party or the other, and are they taken into account in polls?
Thanks for any insight you can provide,
Docpro
Mobile Phone Only Households and Polling
Mark Blumenthal has a pretty comprehensive article detailing an issue that’s been bothering poll watchers: if pollsters only call random landline phone numbers, what’s the impact of mobile-phone-only households on the polling numbers? Blumenthal indica…
Two points:
1) If the choice to go wireless trends together with renting then it also trends together with non-registration, since renters are less likely to be registered to vote (at least at their current residence). That’s at least a reasonable argument that, at current levels of wireless only households, the selection bias due to cell phone use is not very significant.
2) Do Not Call Lists do not apply to polling. Only to commercial solicitation.
turn some states blue (then watch the animations)
http://www.sixdegreesofvoting.com/?ref=3097&mac=bc584b2ae2d4568fef9788e61d9d51a9
Some thoughts…
1. Young people, who usually rent, often go wireless-only. They are more likely to go out when not working or sleeping. They often have roommates: cell phones prevent fights over phone use. They change their residences more frequently than those who have a mortgage, a career, a family, and a couple decades’ worth of accumulated possessions to keep them in one place.
2. Cell phones are more convenient for those whose jobs involve much travel on a daily basis (housekeepers, private tutors, etc.). I’ve no hard evidence, but perhaps such individuals reason that so long as they have a cell phone, they might as well buy a family plan for the others in the household instead of paying for the land line.
3. As Mark Blumenthal mentions, “The geographic distinctiveness of area codes blurs quickly for mobile phones, especially in metropolitan “multi-state” areas like New York City and Washington DC.” Two of my friends have Philadelphia area codes, but live in Stamford and New York. One has a Berkshire County (MA) area code but lives in Boston. Again, this is a result of the high mobility of young people.
Is there a systematic bias of the raw data demographic
information? If cell phones are a problem and if cell
phone-only use skews young, then virtuallly all the
polls should consistently (apart from sampling) have
too few young people. Is that the case? and is this
a significant undersampling?
IF that is the case, then the polls are all off, since
they are excluding a segment of the population. You
can’t reweight for that.
“The early adopters may be young urban renters, but nowadays you see Mexican gardeners and little old ladies with cell phones. As they spread, won’t the mobile phone user profile tend to resemble the population as a whole?”
It doesn’t matter. I’ll show you why.
Let’s say you are trying to sample the number of red, green and blue jelly beans in a large jar by drawing out a handful of fifty. Now let’s say that almost all of the blue jelly beans are out of your reach. Your sample will not be representative, of course. That is the current situation. (Blue = young voters, out of reach = no or unused landline.)
Now let’s apply your argument and say that with time, a lot of the red and green jelly beans filter down out of your reach, too. That doesn’t bring any blue beans any closer to the top, and your handful will still consist of mostly red and green beans with blue beans greatly underrepresented.
My point is that the demographic make-up of what is out-of-reach is unimportant. What is important is the demographic make-up of what is IN reach, and its relationship to the demographics of the whole. It doesn’t matter how many gardeners or little-old-ladies drop their landlines and join the students. As long as the remaining ‘landlubber’ holdouts are still mostly gardeners and little-old-ladies, the representation problem persists.
Paul.
By the way, I work for a market research companies as a telephone interviewer, and I have to say the Caller ID worry is vastly overrated and betrays a basic misunderstanding of the basic reliability of polls. Sure, the personalities of people who obsess over Caller ID might differ and with personalities might differ politics. But that begs the whole questions: even if they do answer your call, how do the personalities differ among people who are likely to agree to participate, versus those who don’t. I work in the polling trenches, and I know that these personalities are markedly different. I can guess with very good accuracy whether that willingness is there from the moment they say hello, from their whole demeanour, from their use (or misuse or unuse) of politeness, and from their tone. Some of these things are related to their daily situation, but a lot is due to personality and the difference between a YES person and a GET LOST person is much greater than any you can attribute to those who screen versus those who don’t.
In other words, polling itself is a suspect activity, and the more you try to pick it apart to find response bias the more you will recognise that it’s bias all-the-way-down and you will finally arrive at the proper entirely skeptical position with regards to polling accuracy, as only a telephone interviewer can truly appreciate.
Paul.
Sorry for the multiple posts, but it’s hard to resist a forum where the daily drudgery of my polling existence actually amounts to some useable knowledge.
Regarding the DO NOT CALL list. We don’t have it in Canada (where I live and work), but the majority of my research calls are in fact to the United States, and I am constantly having to fend off (dozens of times a day!) accusations that I am breaking the law by calling someone who is on this list, and when I explain that it doesn’t apply to polls or research, am I almost uniformly disbelieved. So being on the Do Not Call list does have a big effect. It automatically predisposes the caller to view you as a criminal. The exception for non-soliciting calls is NOT widely understood AT ALL.
Paul.
Cellular Silence, Part 2
A couple of weeks ago, I pointed to a post by John about wireless phones and polling. On Friday, Mark Blumenthal, a professional pollster with his own blog, posted a lengthy discussion on the issue. He does confirm my suspicion…
Cellular Silence, Part 2
A couple of weeks ago, I pointedto a post by John Chase about wireless phones and polling. On Friday, Mark Blumenthal, a professional pollster with his own blog, posted a lengthy discussion on the issue. He does confirm my suspicion…
GOM noted:
People who won’t talk to pollsters — a growing number, I’ll bet.
…My mother and father are voting for the first time this year: my 60+ yr old father recently became a citizen, and my mother – a lifelong US citizen – has finally decided she should start. Her excuse to date had been a fear of jury duty, which is something of a shoddy way out of 40 years worth of civic participation. They’ve registered and figured out where their local polling place is and are in all respects ready to go. They’re in red-state Indiana, however, so they’re more likely to affect the outcome of local races than they are the presidential one.
The other day, in the course of a conversation about politics and polls, my father happened to mention that yes, he actually had gotten telephone calls from some polling organizations. Apparently his response was to hang up… as he found it “INCREDIBLY OFFENSIVE” to receive these automated calls. Had to bite my tongue. There’s one voter who’s not showing up in Indiana poll data.
Myself, I’m cell-only, in Ohio.
Why Wireless Phones Matter
A lot of folks have wondered what impact cell phones have on polls. Mystery Pollster argues that people with only…
Thanks for the information on cell phones and polling.
I have a related question. The number of people who use there computer(internet) on their landline, therefore creating a ‘busy’ signal when a pollster calls. My phone line is usually tied up on the internet.(DSL is too expensive) We use our cellphones for regular calls.
How many people tie up their landlines, and therefore harder to reach?
Another factor I think must be hard for pollsters to quantify is the behavior of (mostly younger) people who are very selective about which calls they answer. I’ve had any number of friends say they were home when I called but didn’t answer because they didn’t recognize the number. I’ve also seen people recognize the number but not answer the phone because they don’t feel like talking to that person.
My parents, on the other hand, wouldn’t dream of not answering the phone, and wouldn’t dream of not talking to the person who called.
How would a pollster control for that?
“How would a pollster control for that?”
They don’t. What most people don’t get is that there are hundreds of such factors that automatically bias any type of polling, and the only things that keeps the result even somewhat intelligible is the statistical inclination for multiple random biases to cancel each other out. In other words, if you conducted a poll on Bush votes vs. Kerry votes, and after each response, you flipped a coin and if it comes up tails, you record the response as the opposite of what the person actually said, chances are the overall results of the poll wouldn’t be too affected. However, it certainly gives the lie to the idea that polling is in any way a revelation of truth, and it helps you understand why pollsters always hedge their bets with a margin of error as well as a ’19 times out of 20′. Hell, I could guess results, claim the same margin of error stats, and thus have a response to any criticism of my results.
Polls aren’t completely meaningless. But they’re pretty close.
“In other words, if you conducted a poll on Bush votes vs. Kerry votes, and after each response, you flipped a coin and if it comes up tails, you record the response as the opposite of what the person actually said, chances are the overall results of the poll wouldn’t be too affected.”
On second thought, this analogy is a bit extreme. The coin flip would basically neutralise the responses and result in an even race, which, it’s true would not affect the results much but only in a basically statistical tie. I hope though that I got my point across … randomly-aligned response pool bias effects do tend to cancel each in general, but you can never know that for sure, just as you can’t be sure that you will actually flip an equal number of heads and tails.
re: land line bias.
I agree w/ the above. I and many of my friends (who tend towards upper middle class) in the bay area have multiple phones, but never-ever answer the land line. Anyone who has a need already has our cell-numbers. And we only weekly or so, if ever, review our land-line vmail (usually just scanning caller-id-s.. also, the vmail greeting encourages callers to send us email – usually our first name with the phone number trailing at a free-mailer.. which is enough to discourage marketers).
This also means we save money by not buying long-distance plans for the “home phone” (cellular rates being more competitive).. so the reason for even bothering with a home phone number is usually you can’t get DSL service without one (in areas w/ lousy cable service), and there are still some financial institutions that like to see a phone number tied to a “place” – and they have explicit checks to prohibit mobile area-codes/exchanges..
Do Cell Phones Make Polling Irrelevant?
“Legal restrictions are preventing political pollsters from reaching millions of Americans this election cycle because they rely exclusively on cell phones,” The Hill reports. “The inability to reach such voters, mainly young people, is contributing to…
unpollable cellphones and kerry
Mystery Pollster had this up a couple of days ago, but the host seems to be a bit testy at present.
Bias Beyond Dispute
The way we would phrase this in biostatistics, is that there is a definite selection bias introduced by excluding mobile-phone-only folks. A bias, of course, is only of consequence if it is differential, if the folks excluded have a systematic tendency to differ from those included in respect to the factor of interest.
I think there’s more here than just a difference in terminology. It is perhaps understandable that political pollsters, stuck as they are with telephone-based polling as the only practical means of finding respondents, would tend to the view that all biases in telephone polling are innocent until proven guilty, not differential unless you have strong reason to think they are. But wouldn’t a reasonable standard of truth be that we don’t know whether a bias is differential or not until we see evidence one way or the other? If we have to wait until exit polls are available for such proof, wouldn’t the most careful practice in the meantime be to report, along with the results of the poll, the magnitude of all these biases, so the reader could form his own opinions of how likely they are to be differential, and at what magnitude?
The point about the relatively small magnitude of this bias, your estimate is 5%, being even further reduced by the differential, which you estimate as about 20%, resulting in a net change of only 1%, is well taken. Even a strong differential, like 20%, doesn’t get you much if your original bias is only 5%. But I understand that the cooperation and contact biases are of much greater magnitude, I have heard as much as 60% combined. With biases of that magnitude, even a small differential could get to a 5-10% net change. Isn’t 5-10% about the magnitude of difference between the aggregate results of the last conventional telephone polls prior to the 2000 election, and the actual outcome?
In biostatistics, the recognized best practice is to report a sensitivity analysis when we are faced with imponderables, such as the size of any differential bias, of sufficient magnitude to potentially change the final result. We report the size of such imponderable factors, any evidence that might guide assigning a magnitude to the differential (and this is where you would think exit polling from the last election would be of help), and then report different calculations of the final result, based on different assumptions. You folks already do a form of this sensitivity analysis when you report results for likely voters, then registered voters, then (less often) eligible voters. In effect, you admit that assuming varying levels of turnout involves a differential bias of imponderable magnitude, but which you can still make educated guesses about. When are you guys going to start being more up front with the bias, which, for all I know as an uninformed consumer, may be larger in magnitude than the difference between eligible and likely voters? Why shouldn’t it be standard practice to report at least the size of the contact and cooperation biases, along with any poll results?
PS: Back to the specific question of the mobili-phone-only bias. If 5% is a reasonable guesstimate for the size of the population that only has a mobile phone, and no land-line, I would think that you should double that, to about 10%, to get a reasonable guesstimate of folks who are functionally not reachable by land-line. Some folks only keep a land line for ADSL conectivity, for their kids only, because their jobs require them to have a land line, for outgoing long-distance calls only, or just out of inertia. So we’re up to 2% net effect.
Another strong assumption, it seems to me, is that we’re not going to see the kind of systematic disenfranchisement that was seen in the election of 2000.
Is there a way for pollsters to express the non-response rate in polls (i.e. a sample of 1200 likely voters with a 3.4% margin of error and a 2% non-response rate)? If so, is there a good reason this information is not published with poll numbers in news stories?
Also, in a sample of 1200, do pollsters find more respondents if they come up short (i.e. 950 respond, find 250 more people willing to answer)? Or do they take the results of the 950 while still referring to the sample as 1200?
Thank you in advance for any assistance.
pl. send me a mobile phone , as i am 62 years old.
m. rashid chudhary
librarian,
beacon house informatics, jinnah super
islamabad
i heard they were bringing out a land line that would also be a mobile pphone ,you gould take the handset out with you when is this going to happen
Mobile Phones and Polling Accuracy
Since I have been posting a lot about mobile phones lately, I wanted to direct you towards a post from last year by Mark Blumenthal, aka Mystery Pollster. If you are interested in polling/statistics at all you should check him out regularly Any…
The trend of dumping one’s land line for a cell phone appears to be increasing, and I am noticing more professionals joining the party. I am hearing more and more discussion of the high cost of maintaining both while many folks’ financial situation is getting squeezed. Clearly, because of this growth, we need to determine its impact on polling, as it most likely is impacting the accuracy of polls.
In an earlier posting, Paul opined that polls aren’t completely meaningless, but they’re pretty close. I think that may be over-the-top as in indictment of polling. What we need are continued assessments of polling accuracy, in light of increasingly difficult factors such as mobile phones and unwillingness to participate. The whole polling enterprise rests on whether they are accurate or not.
This type of sentiment gave Cellufun the idea to introduce polling in our mobile portal at http://wap.cellufun.com. Here is the press release.
http://www.marketwire.com/mw/release_html_b1?release_id=175569