« Newport's Answer to Kurtz | Main | Immigration: Ask Many Questions, Not One »

March 31, 2006

Focus Groups - What They're Not

Yesterday, the Hotline On Call blog reported on focus groups conducted by Republican pollster Frank Luntz recently among Democrats in Iowa and New Hampshire.  Since MP assumes those observations will get noticed in the blogosphere (and since...well... they asked) now is a good time to talk a bit more about the pros and cons of focus groups. 

The Wikipedia has a good explanation of focus groups that includes this basic description: 

In traditional focus groups, a pre-screened (pre-qualified) group of respondents gathers in the same room. They are pre-screened to ensure that group members are part of the relevant target market and that the group is a representative subgroup of this market segment. There are usually 8 to 12 members in the group, and the session usually lasts for 1 to 2 hours. A moderator guides the group through a discussion that probes attitudes about a client's proposed products or services. The discussion is unstructured (or loosely structured), and the moderator encourages the free flow of ideas. Although the moderator is seldom given specific questions to ask, he/she is often given a list of objectives or an anticipated outline.

Client representatives observe the discussion from behind a one-way mirror. Participants cannot see out, but the researchers and their clients can see in. Usually, a video camera records the meeting so that it can be seen by others who were not able to travel to the focus group site. Researchers are examining more than the spoken words. They also try to interpret facial expressions, body language, and group dynamics. Transcripts are also created from the video tape.

In traditional focus groups, a pre-screened (pre-qualified) group of respondents gathers in the same room. They are pre-screened to ensure that group members are part of the relevant target market and that the group is a representative subgroup of this market segment. There are usually 8 to 12 members in the group, and the session usually lasts for 1 to 2 hours. A moderator guides the group through a discussion that probes attitudes about a client's proposed products or services. The discussion is unstructured (or loosely structured), and the moderator encourages the free flow of ideas. Although the moderator is seldom given specific questions to ask, he/she is often given a list of objectives or an anticipated outline.

Client representatives observe the discussion from behind a one-way mirror. Participants cannot see out, but the researchers and their clients can see in. Usually, a video camera records the meeting so that it can be seen by others who were not able to travel to the focus group site. Researchers are examining more than the spoken words. They also try to interpret facial expressions, body language, and group dynamics. Transcripts are also created from the video tape.

MP, like most pollsters, considers the focus group an invaluable tool in the measurement of public opinion.  The great advantage of the focus group is its wide open and unstructured format.  While a survey must follow a standardized structure and fit within tight time constraints, a focus group can be free-wheeling and spontaneous.  Participants can answer in their own words.  If the initial questions are confusing, the moderator can immediately explain or revise or take the conversation in new and unforeseen direction.  Also, the in-person format allows for the moderator to play "show and tell" with video clips, advertisements or new products - something that would be impossible over the phone.   

Traditionally, focus groups have been used most often as a "pilot test" of language, concepts and theories before conducting a formal survey.   Political consultants often use focus groups after conducting formal survey research to pilot test television spots and other forms of advertising.

Another reason for their great popularity, especially in the corporate world, is that research consumers find opinions expressed in a focus groups easier to understand and relate to than numbers in a table or chart.   As Wikipedia puts it, the results are "believable" and have "high apparent validity."

However, that ease of understanding can often lead to misuse.  Thus, it is important to remember the limitations of focus groups, especially the idea that a focus group is not a survey.  To use the research lingo, focus groups are qualitative not quantitative.  That is, they do not allow for projective, quantitative estimates for some larger population.  Put another way, you cannot count answers to a question posed in focus groups (hypothetically, 10 of 20 in an Iowa focus group like chocolate ice cream) and use them to make estimates about the views of a larger population (50% of Iowans like chocolate ice cream).  The reasons are that a sample size of 10 is tiny and more importantly, given the time and travel required, many in the population of interest will lack either the time or inclination to participate.   

Focus group recruiters typically offer a cash incentive (typically $50 to $75 but sometimes much more), a necessary practice that can create its own challenge.  Focus group researchers must deal with the "professional respondents" who would be happy to participate in focus groups several times a week.  (To hear how this process can break down, MP highly recommends the story that aired in March 2002 on the NPR program Marketplace -- the focus group story begins at about 4:10).

Another limitation is the challenge from what researchers call "group dynamic," and everyone thinks of as "peer pressure."  If one highly opinionated participant makes a compelling or emotional argument, others in the group may have a hard time expressing contradictory opinions.  To get around this reluctance, researchers try to keep the demographic composition of groups as homogenous as possible (all female or all white, for example).  They will try to weed out those who might have an "expert" opinion or who typically speak with authority (such as teachers in a group about education issues).  They will also use written exercises to get respondents rooted in their opinions before the discussion starts. 

Unfortunately, the same format that makes focus groups easy to understand also makes them challenging to interpret objectively.  Focus group analysis can be a lot like interpreting a Rorschach inkblot.  It is all too easy to see what one wants to see in a focus group or to make too much out of too little.  That is why most serious researchers recommend following up focus groups with a quantitative survey to try to confirm their apparent findings. 

In the political context, focus groups can also be very sensitive to the kinds of people recruited and the nature of information presented.  Recruit only undecided voters and those unsure about their vote preference, and opinions are likely to change during the group.  Include strong partisans or those strongly committed to a given candidate, and opinions about candidates will be more resistant to change.  Moreover, since a focus group will often expose voters to more political information and discussion in a single evening than they typically experience all year, their reactions can sometimes be highly artificial and deceptive. 

All of which brings me to Frank Lutz's latest focus group project.  It is hard to evaluate the findings since the Hotline report is second hand and includes no information about "size/demographic balance, etc." of the groups.   However, in general terms, what the Hotline report describes is more or less what an internal campaign pollster would do in Iowa or New Hampshire on behalf of a client running for president.  The focus group moderator would first discuss all the candidates, probing for awareness and existing attitudes.  At some point they would probably ask participants to "vote" for their favorite.  Then they would present more information or video-tape of some or all the candidates followed by yet more discussion and possibly a second vote. 

In this case, it appears that Luntz played just one video clip of each candidate (it would be hard to do more given the time constraint).  As such, the reactions he reports depend greatly on the clip he chose to play.  Those clips may or may not provide a decent simulation of the sort of exposure those candidates will get over the next two years.  So while Luntz observations are interesting, it is hard to know what to make of them.  But I know that won't stop political junkies from speculating, so...enjoy!

Related Entries - Focus Groups

Posted by Mark Blumenthal on March 31, 2006 at 01:42 PM in Focus Groups | Permalink

Comments

Isn't Luntz notorious for invoking survey results as sales "proof" for predetermined conclusions: the Contract with America, or the notion that more young people believe in space aliens than Social Security?

Posted by: RonK, Seattle | Apr 1, 2006 1:28:27 PM

A copy of the Luntz paper (pdf, 27 pp.) is posted here:
http://www.dailykos.com/images/user/3/Luntz_2008.pdf

Posted by: Closet Independent | Apr 1, 2006 3:57:25 PM

Playing a clip of each Dem in front of a focus group sounds fine to me. It is not as if these people have never seen Hillary, or Biden, or Warner before. Most of them have their minds made up based on what they have seen or heard about these people already, so a clip might just be a brain refresher.
Luntz was also in California, with a straw poll at the state convention. Even US News and World Report is informing their readers that a 29% for Condi among California delegates (the top rate by the way), shows that if Condi is on the ticket (and I prefer as president) that she could win the state for the Republican party.

Posted by: Pete Larsonf | Apr 1, 2006 7:47:05 PM

I think the latter part is especially true, because the one thing you have said over and over again in this report, is Bush bashing generates applause but not votes. Chances are since Bush is not actually on the ballot in 2008, candidates would probably have to focus a considerable amount on their own platforms.

Posted by: Nadia | Apr 4, 2006 8:11:54 AM

I have no doubt you are correct in your description of Luntz' probable protocol in his groups about the possible Democratic candidates:

"The focus group moderator would first discuss all the candidates, probing for awareness and existing attitudes. At some point they would probably ask participants to "vote" for their favorite. Then they would present more information or video-tape of some or all the candidates followed by yet more discussion and possibly a second vote."


I'd like to suggest, however, that that is a wasteful use of the group. Had I been asked to design a qualitative study about these candidates I would have approached it somewhat differently. Yes, I would have asked about awareness and probed about existing attitudes - that should also serve to identify pros, hijackers and other respondents to keep an eye on - but that shouldn't take more than 15 minutes.

Beyond that, however, I would have focused on particular issues and concerns of the group (or that I would have injected into the group). I would have asked for comments on the relevance and importance of the issues to the participants and then used brief sound bites to stimulate further discussion of each candidate's credibility on the issues. I would be looking for a matrix that compared each candidate's credibility on a given issue to the relevance and salience of that issue to the participants. It seems to me that by deliberately taking commentary out of context the researchers will best be able to develop the kind of issue-oriented hypotheses that can be tested in survey research or on the stump. It also (in the hands of a decent moderator) provides a means for overcoming preconceptions ranging from "Hillary is Satan's spawn" to "Mark who?"

Please note that I am not suggesting that emotional reactions to candidates are irrelevant. Nor am I criticizing Luntz; I've seen him in groups and he's one of the best moderators around. Rather, I am suggesting that emotional resonance to the candidates themselves needs to be balanced against resonance to the candidate's positions on matters of policy. To get a handle on this, moderators need to probe into the relevance and salience of the issues themselves as well as the credibility of the candidate.

The relevance and salience of national security is obviously high, but John Kerry lacked credibility on this issue. Democrats have credibility to spare on a wide range of other issues, but they have not been able to make them either relevant or salient to a majority of voters. On the other hand, plenty of people will say that Bush is a lying cretin - and then vote for him anyway because national security trumps everything else and Bush has more credibility on the issue than any Democrat.

Posted by: ronbo | Apr 5, 2006 5:19:52 PM

Read this:

http://news.baluart.net/article/42/steve-jobs-and-the-focus-group/

;-)

Posted by: Abi | Apr 20, 2006 1:43:39 AM

Why would the Dems trust Frank Luntz when he is notorious for his right-wing leanings and known as a propagandist for the conservative agenda? I think they'd be concerned that he would bias his results toward the Democrat he believes to be the easiest to defeat.

http://kcactive.com/news/newscomm/newscomm2006_06_16.htm

Posted by: AL | Dec 12, 2006 4:53:45 PM

The comments to this entry are closed.