Gallup’s David Moore has posted an analysis this morning (free for today only) with an important lesson on how to write – or perhaps, how not to write – survey questions. The lesson: Double negatives confuse respondents. Put another way, when a question asks respondents if the favor a negative or oppose a positive, confusion (and thus "measurement error") is an inevitable result.
Two weeks ago, on a survey conducted March 10-11, Gallup asked a question that produced an odd result. Nearly two-thirds (64%) of their sample of adults expressed opposition to "a bill that would prevent any foreign-owned company from owning cargo operations at seaports in the United States" (emphasis added)? This result was at odds with a number of other polls showing overwhelming opposition to the Dubai ports deal. Suspecting that the double negative in the question may have confused some respondents, Gallup followed up with an experiment on a subsequent survey (conducted March 13-16) that split their sample into two random halves and asked two slightly different versions of the question.
The first version repeated the question as originally asked (n=502):
Would you favor or oppose a bill that would prevent any foreign-owned company from owning cargo operations at seaports in the United States?
38% favor
58% oppose
4% no opinion
The second question changed the negative phrasing to a positive:
Would you favor or oppose a bill that would allow only U.S. companies to own cargo operations at seaports in the United States?
68% favor
25% oppose
7% no opinion
Thus, changing the words "prevent any foreign-owned company" to "allow only U.S. companies" produced a very different result even though both questions asked the same thing. Moore’s analysis explained the likely reason:
[T]he results suggested that as people were listening on the phone], they may have missed that the [original] question was referring to a bill to prevent foreign-owned companies from owning cargo operations in U.S. seaports. If people were against foreign-owned companies, they would have had to say they were in favor of the bill. That seemed like a double negative from a linguistic perspective, and thus likely to confuse. Also, it was possible that some people might have heard the "foreign-owned companies" part of the question, and not the "bill to prevent" part. So, when respondents said they were "opposed," they may have meant they were opposed to the foreign-owned companies, not the bill itself.
Moore’s analysis speculates that doubt remains about "the exact percentage of Americans who would support the proposed bill," largely because on other questions an overwhelming majority of Americans would support allowing Great Britain to own seaport cargo operations.
It is worth reading in full for that reason, but more importantly for helping to reinforce a classic lesson: Double negatives confuse respondents. Pollsters should avoided them.
This calls to mind the Roper poll from the early 1990’s where they asked “Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” When the question was asked this way about 20% of the people responded that it seeemed possible that the Holocaust never happened. When the question was reworded to drop the double negative the percentage dropped to around 1%.