August 29, 2006
The NYT Reader's Guide to Polls
Jack Rosenthal, a former senior editor of the New York Times, filled in as the guest Public Editor" this past Sunday and devoted the column to a remarkable "Reader's Guide to Polls." The column (which also includes a kind reference to MP's coverage of the AMA's online Spring Break study) provides a helpful sampler of the various sources of imprecision in public opinion polls. It is a worthy general primer, but as with any attempt to condense a complex subject into a very small space, a few items he covered would probably benefit from a more complete explanation.
One example involves the reference by Andrew Kohut of the Pew Research Center to some findings from their polls on gay marriage:
The order of questions is another source of potential error. That's illustrated by questions asked by the Pew Research Center. Andrew Kohut, its president, says: "If you first ask people what they think about gay marriage, they are opposed. They vent. And if you then ask what they think about civil unions, a majority support that."
Those intrigued by that particular finding should definitely download the Pew report from November 2003 that documented the experimental findings. Here's the key passage:
Granting some legal rights to gay couples is somewhat more acceptable than gay marriage, though most Americans (51%) oppose that idea. Public views on giving legal rights to gay and lesbian couples depend a good deal on the context in which the question is asked. On the survey, half of respondents were asked their views on civil unions after being asked about gay marriage, and half were asked the questions in the reverse order. When respondents have already had the opportunity to express their opposition to gay marriage on the survey, more feel comfortable with allowing some legal rights as an alternative. But when respondents are asked about legal rights without this context, they draw a firmer line.
This context difference has little effect on core support and opposition to gay marriage itself, which is opposed by nearly two-to-one regardless of how the questions are sequenced. But opponents of gay marriage are much more willing to accept the idea of some legal rights after they have had the opportunity to express their opposition to gay marriage. The percent favoring legal rights rises to 45% in this context, while just 37% favor the idea alone. Put in other words, opponents of gay marriage are much more likely to accept allowing some legal rights when they have already had the opportunity to express their opposition to gay marriage itself.
Note also that the Pew surveys have shown a modest increase in support for both gay marriage and civil unions in surveys conducted since 2003.
Another topic, brought to my attention by a very alert reader, concerned this passage from the Rosenthal piece:
Respondents also want to appear to be good citizens. When the Times/CBS News Poll asks voters if they voted in the 2004 presidential election, 73 percent say yes. Shortly after the election, however, the Census Bureau reported that only 64 percent of the eligible voters actually voted.
Ironically, as the reader points out, the Census Bureau's estimate of turnout is itself based on a survey, in this case the Current Population Study, which is prone to the same sort of measurement. Professor Michael McDonald of George Mason University produces his own turnout statistics based on aggregate population and vote statistics. His estimate of turnout among eligible voters in 2004 was closer to 61%, not 64%. The Census Bureau explains the imprecision of this particular estimate in a footnote (#2):
The estimates in this report (which may be shown in text, figures, and tables) are based on responses from a sample of the population and may differ from actual values because of sampling variability or other factors.
The "other factor" in this case is the same phenomenon that results in the over-reporting of voting in the NYT/CBS polling -- respondents wanting to be good citizens. I suppose this example teaches that vetting survey results for publication is not as easy as we might imagine.
Speaking of which, the Rosenthal piece also included this news:
The Times recently issued a seven-page paper on polling standards for editors and reporters. "Keeping poorly done survey research out of the paper is just as important as getting good survey research into the paper," the document said.
True enough. But what standards will the Times now apply? If that seven page document has been released into the public domain, I haven't seen it. ABC News puts its Methodology and Standards guide online. Why not America's "newspaper of record?"
"Respondents also want to appear to be good citizens. When the Times/CBS News Poll asks voters if they voted in the 2004 presidential election, 73 percent say yes. Shortly after the election, however, the Census Bureau reported that only 64 percent of the eligible voters actually voted."
While I have no problem believing that some claim to have voted when they didn't there is a possible additional explanation. In my case I did vote in 2004- but not for a presidential candidate. I didn't feel that either deserved my vote so I left it blank but I did vote on a number of state measures as well as for congresscritters, etc.
If asked the question above I would have answered that I did indeed vote. However when counting up how many people voted were they counting total ballots or total presidential votes? If the latter then there would be a discrepency between my statement and the "actual" measurement.
Of course that's probably a very minor case, most people are probably more likely to vote on presidential elections and leave other portions blank.
Posted by: Tlaloc | Aug 29, 2006 9:06:19 PM
I admittedly glossed over the issue you raise. The Census CPS turnout estimate is based on the reported vote cast for president.
If you look at Prof. McDonald's estimates, you'll see that he calculates turnout among the "voter eligible population" two ways: Turnout for the highest office in 2004 (in this case president)was 60.32%. Total turnout was 60.93%. So the difference is 0.61% -- obviously not big enough to explain the gap with the CPS estimate.
Posted by: Mark Blumenthal | Aug 29, 2006 10:32:20 PM
The difference between votes cast (tabulated) and the percentage of people who say they voted could also reflect that some percentage of people who thought they voted didn't have their votes counted. Whether that's due to election fraud, or voter error or equipment error is open to debate, but we know from Florida 2000 that not everyone who attempted to vote has their votes counted.
Posted by: Aaron | Aug 30, 2006 9:26:45 AM
This is a great blog. I'm going to be sure to link yours to mine. Would you mind doing the same for me?
Thank you very much.
Posted by: J. Mark English | Aug 31, 2006 1:56:04 PM
The comments to this entry are closed.