About that YouGov Poll of British Muslims

Reader BK emailed with a question about a survey of Muslims in the United Kingdom conducted for the London Daily Telegraph by the company YouGov just after the first London bombings.  It asked respondents:  "Do you think the bombing attacks in London on July 7 were justified or not?"  Our reader was horrified that 6% of the British Muslims selected the answer "on balance justified" (11% answered "on balance not justified," 77% answered "not at all justified" and 6% were not sure).  BK asks, "Is this a well designed survey? Do you see these results as strong?"

The short answer is that MP is uncertain.  The longer answer provides a good opportunity to talk about the ongoing debate over Internet based surveys.

YouGov conducts its surveys over the Internet, using a panel of respondents that agree to be interviewed at regular intervals for some financial incentive.  According to British polling blogger Anthony Wells, the company has "used advertising and in some cases even specialist recruitment consultants to try and build a panel that reflects all areas of society."  When they conduct a survey, YouGov draws a random sample of its volunteers and weights the results to match the known demographics of the population of Great Britain.

The key issue is that YouGov’s panel is not a probability sample. Probability sampling is the basis for "scientific" polling. Draw a true random sample where every member of the population has an equal probability of being selected (or at least a known probability) and, the results of the survey can be considered projective of the larger population within a statistical "margin of error."  But the YouGov panel (like those used in the US by Harris Interactive and Zogby) are not random samples at all.  They involve hundreds of thousands of volunteers that "opt-in" to the panel through various sources, most often ads placed on web sites. 

Conventional survey researchers argue that polls based on non-probability samples cannot be considered "scientific," that they have no sound theoretical basis.  YouGov, and the other online pollsters, argue that the challenges now facing telephone surveys — especially the response rates that typically plunge below 30% on US surveys sponsored by the news media — undermine the theoretical basis conventional polling.  They argue that with statistical adjustments, their non-probability samples will yield results that are just as reliable as those obtained with conventional methods.

This debate rages among survey researchers.  YouGov has produced notable successes in Britain (see Anthony Wells’ summary), but was way off in its polls of the US presidential election last year.  Their "final prediction" had Kerry beating Bush by three points (50% to 47%), an error comparable to that experienced by the exit pollsters [For a more detailed discussion of this debate, see the paper by the noted academics Morris Fiorina and Jon Krosnick, posted on the YouGov/Economist web site].

All things being equal, MP trusts polls that start as probability samples over those that do not.  In the case of the survey of British Muslims, however, all things are not equal.  Muslims — at 2.7% of the British population — qualify as what pollsters call a "rare population."  That means that trying to survey British Muslims with a standard random digit dial (RDD) probability sample is prohibitively expensive.  To get a sample of 500, the pollster would need to reach a sample of over 18,500 adults and then hand up on all but the 500 Muslims.  YouGov simply sent an email to those in its panel they had already identified as Muslims, and then weighted the results obtained from the 526 that responded "to reflect the Muslim population of Great Britain by age, gender and country of birth." 

Of course, an Internet Panel is not the only way to survey a rare population.  Interestingly, the British survey organization MORI also conducted a survey of British Muslims in mid July.   The MORI online summary says their survey was "conducted on-street and in-home among British Muslims aged 16+" and then, like the YouGov survey, "weighted by age, gender and work status to reflect the profile of Muslims in Britain according to 2001 Census Data."  Presumably, MORI sent interviewers to heavily Muslim neighborhoods where they went door-to-door or stood on street corners, using some sort of random method to select respondents.  Thus, MORI conducted a probability sample, but the same is only representative of the neighborhoods and street corners they sample from. 

MP will not speculate as to which approach is superior.  Neither produces a true probability sample of all British muslims, although no such sample is feasible in this situation.   The two surveys asked very different questions, so we cannot compare the results looking for differences.

One possible advantage of the YouGov Internet poll is that it might have less "measurement error" on a question like the one that troubled our reader.  Ironically, Jonah Goldberg, writing at the NRO’s The Corner wondered about that:

Presumably people who declined to answer or people who shaved their responses did so in order to downplay or conceal their sympathies. I suppose it’s possible that some folks felt pressure from family members to sound more militant thant they are, but I’d have to guess this poll underestimates the problem.

Actually, the fact that it was conducted online probably mitigated the sort of "underpolling" that Goldberg worried about.  Consider:  We know that respondents will often give a less than truthful answer when the truth might create some social discomfort between the respondent and the interviewer.  Of course, the YouGov poll did not involve an interviewer.  Respondents replied by computer.  So in this case, it is not hard to imagine a British Muslim who believed the attacks to be "justified" responding more truthfully to an impersonal web questionnaire than to a person on the other end of the telephone. 

So how reliable is the YouGov survey and what can we make of the results?  MP will leave it up to readers to decide, but urges caution.  The results are interesting and could not have been obtained by other means, but both the YouGov and MORI polls depart from true probability samples.  As such, the results may represent the views of all British Muslims. 

Or they may not. 

PS: Some additional commentary on this poll by Anthony Wells, Gerry Daly and John O’Sullivan.