Disclosing Party ID: Time/SRBI

Legacy blog posts Weighting by Party

As promised, here are comments from Mark Schulman founder and president of Schulman, Ronca & Bucuvalas, Inc. (SRBI) in response to my query about the disclosure party ID results on the Time magazine survey.

I should note that in my original email to Schulman, I wrongly concluded that the Time/SRBI poll "routinely" omitted the results for party ID, because those results were not reported on the release for the most recent survey conducted in January.  In fact, a review of the marginal results from their 2004 preelection studies available for download at the SRBI archives confirms that they regularly included full results for party ID (see the link for "election trend frequencies" at the bottom of each poll analysis).

Schulman clarifies their policy:

Our policy is to post full marginals for each study. Thanks for noting that the demographics are missing from the January survey, just an oversight.  Deadlines are very tight for the Time surveys. We’re often scrambling to get everything done in a very short time. Looks like the demographics didn’t get posted for that survey. I’ll make sure that full marginals are posted for that survey.

SRBI did report party ID distributions for registered and likely voters starting with the early September Time Poll and we continued reporting these distributions through the election. We posted the party data on our releases to anticipate queries about these distributions and in the interest of full disclosure to interested parties, since it had become a contentious issue.

You’ll remember that the party ID weighting issue surfaced with a bang in early September 2005 when we and some other media polls reported major Kerry horse-race slippage.  At least some of the partisan pollsters pooh-poohed the reported slippage, arguing that we should reweight based upon party. They believed that we had somehow oversampled Republicans, hence the Kerry drop. I understand that the reweighting by party ID virtually wiped out the Bush surge…problem solved. I should add that some weighting schemes [used by other pollsters] do attach a minor weight to party ID based upon smoothing the estimation over time.  We don’t do that, but I don’t have a real problem with that, since the impact is minor.

By way of background, our Time polls do collect full demographic data on the entire sample, registered voter or not, and we do weight the entire sample by multiple census demographics, adults in households, and number of phone lines. 

Several of us challenged the party ID reweighting strategy on AAPORNET and on several blogs. In my postings, I warned that reweighting by party ID "can result in serious distortion." In media interviews, my stance was, if you think all is well with the Kerry campaign and that the slippage was just an artifact of ‘too many Republicans,’ then it should be "business as usual" for Kerry. However, if you believe the Kerry drop that we reported, then the Kerry campaign needs to rethink its strategy. (I was grilled on this point in an interview with Air America Radio, for example.)

Schulman also sent along a longer article he posted on the subject of weighting by party ID that originally appeared on the members-only AAPOR listserv in September.  The full text appears after the jump.

Party ID weighting….September 11 Posting on various websites….Mark Schulman

Since we released last week’s poll with the Bush bounce, we’re gotten lots of inquiries about why our poll aand many of the other media polls differ from some of the partisan polls, particularly Zogby, which found little bounce. (I have not actually seen the Zogby poll, but have gotten second-hand reports.) The major reason for the disparity is that most of the media polls, including ours, weight by Census data. Zogby and some others weight on party id. I just penned a response to some academic folks who raised the weighting issue. I’ve attached it below, fyi. Please feel free to discuss.

———–

Weighting by party ID can result in serious distortion of the horserace numbers. Here’s why:

1.  As an observer of party identification tallies day after day on our election surveys, it’s clear that we’re not measuring a constant factor. It varies slightly, sometimes even significantly, day by day, week by week. 

2. Why does it vary? Most polls place the party ID question near the end of the questionnaire, so that it does not interact or contaminate the horserace measure and any other head-to-head candidate comparisons. David Moore has an excellent piece on the Gallup web site discussing the likely impact of question order on party ID measurement. The horserace always takes priority over party ID in question order, since that’s the topline number we report. As a result, respondents may tend to bring their party ID in line with their partisan choice, particularly after having gone through an extensive battery of election items. It’s simply "cognitive consistency." Hence, a Bush surge, for example, might elevate the number of voters later in the survey identifying themselves as Republicans.

3. Since party ID is a "variable" and not an enduring constant, as is age or gender, it varies!

4. Voting behavior literature from the 1950s and 1960s (Campbell, Converse, Miller and Stokes, The American Voter, for example), used to posit party ID as anchoring partisan choice, as if it were a constant. It’s likely that party ID was a more enduring "constant" in the 1950’s, but, that was then, and this is now. Voters are just not as tied to party as in the past. Let’s get over this likely out of date notion that party ID is a constant that anchors the vote. The causal arrows here are unclear, which influences which? We can construct several models of party ID as both a dependent and an independent variable.  The traditional model posits party ID as an independent variable.  We now see it likely as both an independent and dependent variable, with all sorts of interactions.

5. Hence, weighting by party ID, and it’s party ID, not party registration, can seriously distort the horserace data. Weighting by party ID would damp down the Bush surge over the past few weeks. Yes, there may be some "at home" selection bias when we interview during party convention periods. However, not all that many folks watch the conventions and the networks provide little convention coverage.

Finally, my choice, and the choice of most the major media polls, is to weight by factors that we know are real, such as age, gender, region, education, number of adults in household, number of voice phone lines, etc. While you can argue about the reliability of Census data, I’ll place my bets with the Census rather than party ID.

David Moore has a good discussion of this issue as well on the Gallup web site.

That’s the short version of my views. I really do believe that we need to put this issue to rest and stop pretending that there’s legitimacy to party ID weighting. I look forward to further comment!

Very best wishes,
Mark Schulman
Schulman, Ronca & Bucucvalas, Inc.

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.