Rasmussen Update: A Lesson in Measurement Error

IVR Polls Legacy blog posts Measurement Issues

Although still playing catch-up on the “day job,” I want to highlight something that has appeared without fanfare over the last week on Bush job approval page on the Rasmussen Reports website.  Rasmussen, as most regular readers know, conducts surveys using an automated methodology that asks respondents to answer questions by pushing buttons on their touch tone telephones.  I have looked closely at the Bush job rating as reported by Rasmussen and noted that their surveys report an approval percentage that is consistently 3 to 4 percentage points higher than the results of other national surveys of adults (see the Charles Franklin produced graphic below).   Last week, Rasmussen offered this explanation: 

When comparing Job Approval ratings between different polling firms, it’s important to focus on trends rather than absolute numbers. One reason for this is that different firms ask Job Approval questions in different ways. At Rasmussen Reports, we ask if people Strongly Approve, Somewhat Approve, Somewhat Disapprove, or Strongly Disapprove of the way the President is performing his job. This approach, in the current political environment, yields results about 3-4 points higher than if we simply ask if people if they approve or disapprove (we have tested this by asking the question both ways on the same night). Presumably, this is because some people who are a bit uncomfortable saying they “Approve” are willing to say they “Somewhat Approve.” It’s worth noting that, with our approach, virtually nobody offers a “Not Sure” response when asked about the President.

Although I  had not considered this possibility before, Rasmussen’s finding makes a lot of sense.  In my own experience, the word “somewhat” (as in somewhat agree, somewhat favor, someone approve, etc.) softens the choice and makes it more appealing. 

It is worth noting that virtually all of the other national public pollsters begin with a question posing the simple two-way choice: “Do you approve or disapprove of the way George W. Bush is handling his job as president?”   A few then follow-up with a second question, such as the one used by the ABC News/Washington Post poll:   “Do you approve/disapprove strongly or somewhat?”  AP-IPSOS uses a similar follow-up, as does LA Times/Bloomberg, Diageo/Hotline and Cook/RT Strategies

The key point:  On the Rasmussen surveys, respondents choose between four categories:  strongly approve, somewhat approve, somewhat disapprove and strongly disapprove.   On all of the other conventional surveys — including the one listed above that later probe for how strongly respondents approve — respondents initially choose between just two categories:  approve and disapprove.**   They will only hear the intensity follow-up (strong or somewhat) after answering the first part of the question.   

So except for the pollsters (Harris and Zogby) that use entirely different answer categories (excellent, good, fair or poor), Rasmussen is the only national pollster I am aware of that presents an initial choice involving more than “approve” or “disapprove.”  As Rasmussen argues, the consistently higher percentage he gets for the Bush job approval rating may well be an artifact of the way he asks the question.   If so, the difference in the graph above may be due to what pollsters call “measurement error.”

I hope Scott Rasmussen will consider releasing the data gathered in their experiment that “tested this by asking the question both ways on the same night.”   This sort of side-by-side controlled experimentation (where the pollster randomly divides their sample and asks slightly different versions of the same question on each half) is a great example of the scientific approach to questionnaire design and survey analysis.  We would all benefit by learning more.

**All of the pollsters mentioned above allow for a volunteered “don’t know’ response in some form, although the procedures that determine how hard live interviewers press uncertain respondents for an answer may vary among polling organizations. 

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.