September 29, 2004
Why & How Pollsters Weight, Part I
I want to return to the issue of weighting, beginning with a question commenter Ted H:
What puzzles me about the idea of weighting by party ID is that surveys are supposedly designed to sample from the population randomly. This assumption is also the basis of the margin of error they calculate. So if you sample randomly but then adjust the results because you didn't get the percentages of D and R that you expected, you have thrown out the design of the survey.
Unfortunately, as many of you guessed, perfect random samples are impossible under real world conditions. Although we begin with randomly generated telephone numbers that constitute a true random sample of all US telephone households, some potential respondents are not home when we call. Others use answering machines or Caller ID to avoid incoming calls, and increasingly large numbers simply refuse to participate. All of these unreachable respondents create the potential for what methodologists call "non-response bias.” (This is a major topic for further discussion – if you can’t wait, this review of the vexing issue by ABC Polling Director Gary Langer is one of the best available anywhere).
Telephone surveys also exclude the small percentage of households lacking home telephone service. Unfortunately, this percentage is growing rapidly due to the number (estimated at 2-5%) that has switched off their wired phone service altogether in favor of cell phones (another important topic that I am purposely glossing over for now – we’ll definitely come back to it).
These missing respondents can cause error – also known as bias – only if the missing respondents are different from those interviewed. Pollsters weight data as a strategy to reduce observed bias.
Pollster strategies for weighting seem to fall into three general categories. I’ll take up the first tonight and discuss the others in subsequent posts over the next few days.
The first involves the classic strategy used by most of the major national media surveys, including CBS/New York Times, ABC/Washington Post,* Gallup/CNN/USA Today, Newsweek, Time, The Pew Research Center and the Annenberg National Election Survey among others. They begin by interviewing randomly selected adults in a random sample of telephone households. Even if they ultimately report results for only registered voters, they ask demographic questions of all adults. They then typically weight the results to match the estimates provided by the U.S. Census for gender, age, race education and usually by some geographic classification. This weighting eliminates any demographic bias, including chance variation due to sampling error.
The key point is that they weight only by attributes that are fixed at any given moment, easily described by respondents and matched to bulletproof Census estimates using language that typically replicates the Census questions.
Finally, to be clear, none of these organizations weights by party! Contrary to what I have seen written elsewhere neither CBS/New York Times, ABC/Washington Post,* Gallup/CNN/USA Today, Time, Newsweek, The Pew Research Center and nor the Annenberg National Election Survey weights their results by Party ID. [UPDATE: ABC News does weight its October tracking survey of likely voters by party (but not registerd voters and not surveys prior to October)]
What about CBS? I’ve read several posts – including one on the comments section of this blog – that repeat the myth that CBS News weights by party. The proof? The table below (which I reproduced from the last page of this release) which shows weighted an unweighted samples sizes for Democrats, Republicans and Indepedents. CBS provides this data because their report includes tabulations by party for each question, and they are providing disclosure of their sub-sample sizes exactly as required by the disclosure standards of the National Council for Public Polls. They go even one step further, providing unweighted counts to assist in those who want to calculate sampling error.
|Reg. Voters –- Republicans||343||323|
|Reg. Voters –- Democrats||324||332|
|Reg. Voters –- Independents||264||246|