While we’re on the topic of poll questions commissioned by web sites on the blogosphere’s left wing, here is a truly innovative development: The blog known as MyDD has decided to go beyond simply placing a few questions on a larger omnibus poll and is now fielding the first of a “semi-annual series of netroots commissioned polls” in collaboration with its readers that seeks to ask questions that mainstream news polls “seem unable to ask themselves.”
Here are the details. The project is the brainchild of MyDD blogger Chris Bowers, a name that should be familiar to regular MP readers because of his frequent commentary on mainstream media polling. Bowers was a vocal critic of the Gallup likely voter model during the 2004 campaign, and shared his concerns in a speech (delivered via video) to a roundtable at last summer’s conference of the American Association of Public Opinion Research (AAPOR). MP may not always agree with Bowers conclusions, but I find his comments to be typically thoughtful, constructive and well intentioned.
Although polls sponsored by partisan groups are nothing new, the MyDD project has taken a unique approach by involving its readers not only as sponsors but as participants in the process of drafting questions. A little over two weeks ago, Bowers announced the project along with a rough draft of questions he had in mind (mostly on Iraq, domestic spying and impeachment) and then invited readers to “critique these drafts, and to offer your own questions for possible inclusion.” That first draft received more than seventy comments. Three days later, Bowers posted an updated draft of questions and again invited comments and critiques. This post drew another 50 or so comments, including a critique from occasional MP correspondent “Professor M.”
Meanwhile, Bowers and his colleagues hired a professional pollster – Joel Wright of Wright Consulting – partnered with a non-profit and made several appeals to raise the roughly $16,000 to cover the cost of fielding the survey. The most recent update, posted on MyDD by Wright himself (blogging under the nom de Internet “Sun Tzu”), indicates that the poll is now in the field, and they expect calling to be completed by Tuesday or Wednesday night.
MP has been kicking himself for not noticing the MyDD polling project until last week, after they finalized the questionnaire and put the survey into the field. Had I noticed it earlier, I might have made a few suggestions about the questions, but more importantly, would have invited MP’s knowledgeable readers to do the same. You may wonder, as I do, what questions (and what language) Bowers and the others at MyDD settled on. For example, the second draft posted by Bowers included essentially the same Zogby impeachment question I critiqued here last week. I emailed Bowers last week, and he let me know that the final version differs from Draft Two and the Zogby version of the impeachment question had been dropped.
Bowers and Wright obviously prefer to hold the final version of their questionnaire until the conclusion of the survey. That seems fair given that no mainstream media poll I am aware of discloses in advance when it is fielding a new poll, much less the wording of the questions they plan to ask. Given Bowers’ passionate commitment to greater transparency in polling, I think we can expect an unusual level of disclosure in the days and weeks ahead.
It will be interesting to watch. Whatever your views of MyDD’s politics, we should thank them for providing, at very least, a great new opportunity to learn something about the survey process.
PS: Happy Birthday Chris.
Early prediction: The impeachment question will be worded in such a way as to guarantee a high “yes” response. IMHO, the only fair question to ask would be something like, “Based on what you know now, should President Bush be impeached?”
The formulations that were discussed in the second draft are a sophisticated form of push-polling. “If it were determined that President Bush was involved with domestic spying that was illegal, do you think Congress should hold him accountable through impeachment?” Of course, the “should hold him accountable” is steering those polled towards the “right” answer, and the If/Then format also encourages more yes responses.
There’s a reason why they don’t ask the question point blank in my formulation; because it won’t get the desired result. Bowers may have a passionate commitment to greater transparency in polling, but I suspect he also has a passionate commitment to getting Bush impeached.
> Although polls sponsored by partisan groups are nothing new
You saying sponsorship means the results aren’t reliable?
I sent a recent American Research Group poll information regarding unfavorable ratings of Bush to a Republican friend. He said ARG were small timers and had no credibility.
I was wondering whether you or any place provides a profile of each pollster and their general reputation. Is ARG with any credibility or is it because the unfavorable ratings spurred him to say it is. I would rebut him but frankly I have no idea.
Contrary to what Brainster seems to think, a question you don’t like–even for good reason–isn’t NOT the same as a push poll. A push poll is not a poll at all. It’s a campaign message DISGUISED as a poll.
What Brainster is really talking about is question bias–a serious, but quite different issue. All questions one could ask about impeachment would carry some sort of bias. The one Brainster prefers is one that reinforces the media bias that has been suppressing plain old-fashioned fact-based coverage of Bush’s transgressions.
Its a good question to ask, but not inherently better than the one he objects to. Both give different perspectives on the same underlying issue. There is no one “right” question to ask. We learn more from a variety of different questions with different slants than we do from some mythological “perfect” question.
That said, both questions could easily be improved. Brainster’s question could be improved by being more specific: “Based on what you know now, should President Bush be impeached for approving domestic spying without a warrant?”
The question he objects to could be improved by providing an alternative, such as: “If it were determined that President Bush was involved with domestic spying that was illegal, do you think Congress should hold him accountable through impeachment? Or would that be too extreme a response?”
Paul, I agree that it’s not really push-polling as that term is known; question bias is probably a better term.
But I disagree that specificity is helpful. Of the many calls for impeachment that I have seen on the web, most mention many other grounds–extraordinary rendition, “stealing” the election(s), etc. So why limit it to one issue? Better still, why not ask my “Should he be impeached?” question, and if they say yes, ask for the reasons?
I still think the “hold him accountable” portion is steering the respondent to the desired answer.
brainster,
Considering that I assume you agree with the idea of “innocent until proven guilty” then I’m surprised that you think an unqualified question such as “Should he be impeached?” would more accurately gauge the temperament of the populace. More likely that would give us the same results as a self identification polls.
I would guess that most Americans are not likely to want to indict or certainly convict without even being given the charge. That reasonable expectation is related to some vague American value that I’m sure is written down somewhere.
Couching the question as an if/then is probably the only way to ask the question …as long as the exact question is also published with the poll results.
What is the goal of the polling on the impeachment question? Is it legitimately to gauge public opinion on impeachment? Is it to “prove” a groundswell of public opinion against President Bush? Is it to determine whether impeachment’s a “go” on the “domestic” spying issue? Is it to determine if it’s a “go” to “consider” impeachment hearings?
Yes, I know, innocent until proven guilty. I will believe that when I see posts headlined “Consider Holding Hearings as to Whether Bush Should Be Impeached!”
Like lawyers sometimes say, “Don’t ask a question if you don’t already know the answer or don’t want to know the answer.”
I guess you are making the argument to join the latter group.
CS, that was the point of my prior comment. What is the goal? If it’s to gauge public opinion on the impeachment of Bush in general, then I think you want as general a question as possible that hones in on impeachment. Throwing in terms like “consider” and “hold him accountable” run the risk of giving you a false reading.
So you would prefer:
“Should Brainster push a child to the ground?’
as oppsed to:
“Should Brainster push a child to the ground to save her from being hit by a runaway truck?”
I agree that wording is important and that that particular question’s wording wasn’t clear enough but you seem to be arguing against clearer wording as well. It is too easy to debunk your alternative because the results will likely bring the expected partisan answers as opposed to thought through answers.