We have had some new developments over the last few days regarding the online Spring Break study conducted earlier this year by the American Medical Association (AMA). The story, as long time readers will recall, involved an AMA release that initially misrepresented the study, calling it a "random sample" complete with a margin of error and implying in some instances that results from a small subgroup of women that had actually gone on Spring Break trips represented the views of all the women in the survey. While my posts on the subject received a fair amount of attention in the blogosphere, the mainstream media — including outlets that had reported the misleading survey — largely ignored the controversy. This week that changed.
Here are details and links:
Although I had missed it, the New York Times did make a formal correction of a Week in Review story that cited results of the poll soon after American Association for Public Opinion Research (AAPOR) President Cliff Zukin wrote the Times to complain. Their correction now appears at the end of versions of the story available on the Web or through a Nexis search:
For the Record
A chart on March 19 about the history of spring break referred incompletely to an American Medical Association survey of female college students and graduates on vacation behavior. It was conducted online and involved respondents who volunteered to participate; they were not chosen at random.
Earlier this week, the Washington Post‘s Howard Kurtz devoted his Media Notes column to the story. Kurtz reviewed some of the most colorful headlines and quotations from the initial media coverage. "At the risk of spoiling the fun," he concluded, "it must be noted that this poll had zero scientific validity."
Kurtz also quotes Richard Yoast, the director of the AMA’s Department of Alcohol, Tobacco and Other Drug Abuse as saying,
[H]is organization posted a correction on its Web site to note that this was not a nationwide random sample and should not have included a margin of error, as in standard polls. "In the future, we’re going to be more careful," he says.
While they are at it, the AMA might want to be a bit more careful about the way they post corrections. As noted in my original post on this subject, the AMA did correct the methodology blurb in their online release, but the corrected version includes neither a trace of the original misrepresentation nor any statement that the current version corrects the original. Also, as Kurtz points out, the corrected AMA release continues to highlight statistics based on "only the 27 percent of the 644 respondents who said they had actually been on spring break," yet still "make[s] no distinction between those who have taken such trips and those who haven’t" (see this post for details).
The appearance of the Kurtz item may have been the reason that the Associated Press issued this correction just yesterday:
Correction: Spring Break Risks story
Eds: Members who used BC-Spring Break Risks, sent March 7 under a Chicago dateline, are asked to use the following story.
05-31-2006 15:23
CHICAGO (AP) _ In a March 7 story about an American Medical Association survey on spring break drinking and debauchery among college women and graduates, The Associated Press, using information provided by the AMA, erroneously reported how the results were obtained. The AMA now says participants were members of an online panel, not a random sample
Finally, today’s Numbers Guy column by the Wall Street Journal‘s Carl Bialik takes a close look at the story and the new communications initiative that AAPOR will undertake to try to react to stories like this more quickly:
Sixty years after its founding, a key association of professional pollsters is dismayed with all the bad survey numbers in the press. In an overdue response, the group is seeking new ways to curtail coverage of faulty research…
"Our ability to conduct good public opinion and survey research is under attack from many sides," the group’s long-range planning committee wrote in a May report. As part of its response, Aapor, as the group is known, plans to hire a staffer to spot and quickly respond to faulty polls.
If Aapor does come down hard, and quickly, on bad research, it could drive pollsters to do better work and disclose their methods more fully, and perhaps even introduce higher standards to what is today an unruly industry. However, a solitary staffer will be hard-pressed to improve the treatment of polls by a numbers-hungry print and electronic press. [link added]
[Interests declared: I am an active AAPOR member and was recently elected to serve as a two-year term as associate chair and char of AAPOR’s Publications and Information Committee, and as such will help oversee AAPOR’s new communications initiative].
Bialik’s column is worth reading in full, as always, and Bialik and his editors are to be commended for his thorough and thoughtful treatment of this complex subject. However, as long as we are on the topic, the Journal‘s editors might want to consider some questions about its own reporting of online surveys:
- If it was appropriate for the AMA to quickly correct the "troubling" assertion that its online study had a margin of error, why does the Journal continue to report a margin of error of "between 3.2 and 4.3 percentage points" for the online Zogby statewide polls that it sponsors (see the "methodology" link)?
- Regarding the surveys conducted for the Journal by Harris Interactive, why does this March poll story refer the survey as a "telephone poll" in the third paragraph, but as an "online survey" in the methodology blurb? And why did the Journal‘s Harris tracking polls shift from consistently labeled "online surveys" in January and February, to those identified as "telephone polls" in April and May? And why do the tables in the April and May stories compare results across all of these studies without any reference to the apparent switch from the online mode to the telephone?
Perhaps more corrections are in order.
Hi Mark
I love your blog and have been following it since you helped decode the elections for me in 2004. Long-time reader, first-time commenter, etc. 🙂
This was a terrible study for a number of reasons. However, it strikes me that you and AAPOR are coming down rather hard on the issue of random panel sampling versus true ‘random’ telephone sampling. In fact, the position you’ve taken here makes me wonder whether the vast majority of polls ought to be published with a margin of error at all.
The purist position, as I understand it, is that samples selected from a panel can only ever be a convenience sample because they are made up of volunteers, however selected. A random sample, by contrast, includes respondents who have not volunteered for research so therefore can be called a truly random sample of the population. Only a truly random sample can be projectable and thus only a truly random sample should report a margin of error.
To me, the purist position depends on an idealized vision of sampling that is rarely actualized.
In particular, I think the purist position runs aground somewhat when we consider the decline in telephone survey cooperation rates, which for many polls is often in the single digits.
If we don’t have 100% cooperation rates to our survey invitations, however sent, then our sample is by definition made up of volunteers who happen to be selected and available for that particular study at that particular time. Whether the volunteers come from RDD phone sample or panel sample, they are volunteers nonetheless.
I question whether much or any polling work that is done truly meets the AAPOR standards, fully considered.
How do you handle these questions in your practice?
…of course, you are correct saskfan about the scientific ‘requirement’ for a truly random-sample.
The validity of all statistical sampling rests on that core foundation.
Without a ‘random’ sample {… all individuals in the defined population have an ‘equal’ probability of being ‘sampled’} — there is ‘zero’ scientific validity of the results.
But true random samples of ‘opinion’ from a large population of humans — are almost impossible to get.
Endless subjective fudge-factors are used by pollsters to try and compensate for their lack of a random sample — but the absence of that random-sample always means the absence of ‘scientific’ validity in the poll.
All mainstream media ‘polls’ should therefore be labeled as “estimates” … rather than ‘scientific measurements’.
_________
Mark —
It’s great that the NYT corrected its erroneous report about the Spring Break poll, but it’s unfortunate that the paper chose to bury the correction in its “For the Record” space. According to a story in Editor & Publisher from October 2004 concerning the NYT’s corrections policy,
“Substantive errors,” the editors said, will continue to fall under the Corrections heading. These are mistakes that “have materially affected the reader’s understanding of a news development.” Narrower errors involving, for example, spellings, dates, or historical references, will appear under the heading, “For the Record.”
Based on the above one should not expect to find substantive corrections in FTR. Unfortunately, however, the Times has an unhealthy habit of correcting substantive errors that are small but embarassing in FTR rather than Corrections.
The Times also has a unique, self-serving quasi-exculpatory formulation for many of its mistakes: As in the spring break correction, the NYT often says it “referred incompletely” to particular facts. Often, as here, this is like incompletely referring to black as white.
I mention this only because no media outlet deserves the moral high ground regarding corrections when it is as lacking in transparency as the Times. The Times also seems to have particular difficulty with data, especially polling data. Unfortunately, it is no longer possible to differentiate innumeracy from dishonesty.