Weighting by Party

Legacy blog posts Weighting by Party

It is nearly impossible to blog on political polling this week without discussing the subject of weighting by Party ID.  Ruy Teixeira of the Emerging Democratic Majority and Chris Bowers of MyDD have been arguing forcefully (here and here) that major polls are oversampling Republicans.  Yesterday, Mickey Kaus posted a retort by an emailer ("Y”) who described the call for weighting as the "new, cocoon-building liberal analyst meme.”

I have a lot of respect for Ruy Teixeira, and I think his critiques of Gallup Poll’s likely voter model have merit.  However, on the issue of weighting, I agree with Kaus and "Y” (whose post is worth reading in full).

Let me add a few thoughts. Party Identification is one of the longest tracked and closely examined questions in political polling. "Generally speaking, do you consider yourself a Republican, a Democrat, an independent or what,” has probably been asked on more surveys over more years more consistently than any other question. 

The most important thing to remember is that Party ID is an attitude, not a demographic.  People can change their views of political parties.  They cannot change their age, gender, race, years of education and locale (unless they’ve moved). 

However, for many years political science students were taught that Party ID, like religious affiliation, rarely changes among adults.  This belief was the result of two famous panel (or "longitudinal”) surveys conducted by the University of Michigan’s National Election Studies (NES) that interviewed the same respondents three times over a four-year period.  These studies convinced researchers that Party ID was highly stable at the individual level. 

One limitation of those studies was that they were conducted during periods (1956-1960 and 1972-1976) when overall levels of Party ID remained virtually unchanged.  Later closer examination of the data showed that even then, one of every four respondents had changed their answer to the basic Party ID question over the span of four years.  Most of the change involved shifts between independence and one of the political parties (See Samuel L. Popkin, The Reasoning Voter, pp. 53-55).

Partisanship has weakened significantly over the years. The percentage of adults that describes itself as independent has grown sharply (from 23% in 1960 to 36% in 2002, according to NES). 

Moreover, more recent studies have shown evidence of significant short-term change in Party ID. The 2000 Annenberg National Election Study (NAES), like the 2004 study now underway, was a daily tracking survey that ultimately included more than 58,000 interviews over the course of the year, roughly 5000 interviews per month.  NAES observed that the percentage of the electorate identified as Independent "was not stable over time.”  In a chart on page 61 of Capturing Campaign Dynamics, Daniel Romer and his colleagues showed the percentage of Independents falling steadily from roughly 31% to 27% during the conventions, then spiking 8 points to 35% just after the Democratic convention in early September, then falling off again steadily back to roughly 28% on election day, then plummeting sharply to below 25% a few days later.  No surprise that they concluded:

Surveys that are weighted by party identification may be operating under some misconceptions about party identification.  Party identification may not be as stable as once thought and could be considered an indicator of the respondents’ attitudes toward candidates at a given moment of the campaign. (p. 61)

There is another good reason to be weary of automatically weighting survey results by Party ID.  Most of the national polls ask Party ID toward the end of the survey.  Campaign pollsters can tell you that Party ID can vary with the content of the questions that precede it.  Include a long battery of items on the health care or the environment and you are likely to get more Democratic identifiers by survey end than if you include a long battery on fighting terrorism. 

I am not arguing that a pollster should never weight by party – there may be times and conditions when it is appropriate (I’ll try to suggest a few in a future post).  But I’m very wary of automatically weighting every survey by the results of a four-year-old exit poll, given everything we know about the potential for short term variability of Party ID.

I am also not arguing that the Republican bump in Party ID seen in the Newsweek, Time, CBS News and ABC/Washington Post polls is likely to persist.  Nor do I think George Bush has a massive lead.  I agree with Ruy and Chris (and Mickey for that matter) that the race is far from over, that Kerry can still win.  The post convention bump may have already faded.  I hope it has.  But I don’t know that it has.  And that’s the point. 

My first boss, Harrison Hickman, used to say that a pollster’s job was to bring reality to the table.  Whether we work for a news outlet or a political campaign, our task is to describe voter attitudes as they are, not as we might wish them to be. 

Is that what is happening here?  Consider this.  Teixeira says in a post yesterday that recent polls showing a "sudden tilt” toward a Republican advantage of 4-5 points or more in Party ID are implausible, when past exit polls and national surveys have typically showed a 3-4 point Democratic edge. 

Just three months ago a poll by the Los Angeles Times showed a similarly abrubt "tilt” in party ID (to a 13 point, 38% to 25% Democratic advantage) that helped give Kerry a seven point lead.  Matthew Dowd, Bush’s pollster argued that the LA Times Poll was "a mess” because it had not been weighted by Party ID.

And what was Teixeira’s take?  He defended the LA Times Poll.  Money quote:

There are ample grounds for thinking there is, in fact, a surge toward the Democrats and their positions and away from the Republicans and their positions among the broad electorate. A growing Democratic party ID advantage is a logical consequence of that surge, since party ID does not remain stable as political conditions change….Conclusion: there is no good reason to ignore the results of this poll (unless you’re Matthew Dowd, of course, who has his own reasons for doing so).

Apparently Party ID isn’t the only thing that can change in a few month’s time.

[Continue reading about Weighting by Party]

Mark Blumenthal

Mark Blumenthal is the principal at MysteryPollster, LLC. With decades of experience in polling using traditional and innovative online methods, he is uniquely positioned to advise survey researchers, progressive organizations and candidates and the public at-large on how to adapt to polling’s ongoing reinvention. He was previously head of election polling at SurveyMonkey, senior polling editor for The Huffington Post, co-founder of Pollster.com and a long-time campaign consultant who conducted and analyzed political polls and focus groups for Democratic party candidates.