The Associated Press, in its usual partnership with IPSOS, released a survey yesterday (story, results & methodology) on attitudes on today’s holiday honoring Martin Luther King, Jr. and his dream of racial equality:
Three-quarters of those surveyed say there has been significant progress on achieving King’s dream. But only 66 percent of blacks felt that way…
Only 23 percent of respondents say they will do anything to commemorate the national holiday that took effect in 1986 after a lengthy campaign in Congress to honor King. A solid majority of blacks, 60 percent, say they will get involved in holiday activities.
If you do nothing else to commemorate the holiday, read all of Will Lester’s AP story on the poll and its implications.
To allow for a sufficiently large sample of African Americans, AP-IPSOS conducted an "oversample." That term often confuses readers and deserves a bit more explanation. The AP methodology summary tells us that they interviewed 1,242 adults, including 312 blacks and adds this sentence:
The total sample includes an oversample of blacks, completed in part by interviewing people who self-identified as black in previous Ipsos telephone surveys.
Do you find that explanation a bit confusing? If so, here’s how it works: AP-IPSOS typically conducts a representative sample of about 1,000 adults. I am going to assume they did the same on this poll, and guess at the other numbers involved. A base sample of 1,000 would have included a representative sample of African Americans. In this case it appears to have been roughly 70 interviews.** Media pollsters typically shy away from reporting on subgroups that small since they have very large statistical sampling error (for n=70 it would be at least 11%). So they sometimes conduct an "oversample" of additional interviews among respondents in the subgroup of interest to increase the sample size. In this case AP-IPSOS conducted (again, my guess) roughly 242 additional interviews among "people how self identified as black in previous IPSOS telephone surveys."
In order to report results among all adults, the pollsters weight the total pool of interviews (n=1,242 in this case) so that the proportion of African Americans in the weighted sample matches Census estimates for the U.S. Population (roughly 11%). This is an important point: Results reported for "all adults" in this AP-IPSOS poll come from a statistically adjusted sample that "weights down" the African-American oversample to its appropriate size in the U.S. adult population.
Most pollsters will also use Census estimates to weight on other demographic factors to correct for small differences resulting from sampling error or non-response bias. In this case, AP-IPSOS tells us that they also weighted by other factors "such as" age, sex, region and income."
**Again, if I’m guessing right, the AP-IPSOS base sample included roughly 70 interviews, or 7%, among African Americans. A perfectly representative sample of U.S. adults would have been roughly 11% African-American. Why the (apparent) difference? Response rates tend to be lower in urban areas, and as a result, unweighted national samples typically under-represent African Americans. As noted here previously, most national pollsters typically weight African-Americans up slightly to match US Census estimates.
Poll: Most See Significant Racial Progress
Most Americans believe there has been significant progress in achieving Martin Luther King Jr.
Hmmm. Polling firms keep track of the phone numbers and characteristics of their respondents so that they can call them again when they need an oversample.
Since out of the theoretical 1000, there were 70 in the subgroup of interest, one would assume that there would have been more than 242 in the other polls that have been tracked and kept track of. So then the question is what methodology was used to get the 242. Were they selected (perhaps based on the recentness of the poll or the outcome of the poll or even their responses within a poll), was all past participants who fit the criteria contacted and this is what came out, were they predominantly the urban which MP thinks are the underrepresented.
And if a group is underrepresented based on non-response, then what does that make those who self identified with the group in the previous polls. Since the ones answering aren’t typical (they are answering when the group norm is to not repond to the poll), are we seeing the activists, the lonely, the higher or lower educated of the group, etc. It all doesn’t give me a warm fuzzy about the poll.