Rasmussen Adopts Dynamic Party Weighting

As promised here on MP, pollster Scott Rasmussen yesterday announced a new party weighting procedure for his automated tracking survey:

Beginning today (Sunday), Rasmussen Reports Job Approval updates are based upon data using a slight modification to our targeting and weighting process. From this point forward, we will set our partisan affiliation weighting targets based upon survey results obtained during the previous three months. These shift only modestly month-to-month, but the change could be significant over a long period of time.

Based upon the past three months, the current targets are 36.6% Democrat, 33.5% Republican, and 29.9% Unaffiliated. These targets will be updated monthly. Previously, our weighting targets assumed an equal number of Republicans and Democrats.

We have adopted this system because we believe it allows us to maintain the day-to-day stability needed to follow trends while adjusting periodically for any substantive shifts in partisan affiliation (see trends in party affiliation).

The practical impact of this revision is modest in the current environment. The new approach will result in the President’s reported ratings being a point or two lower than they would have been under the old system. Today’s reading [of 38% job approval] would have been 39% using the old approach.

Thus, visitors to Rasmussen Reports will now benefit from the following new features:  Regular disclosure of the party weight target and the knowledge that Rasmussen is weighting to reflect a recent overall average, not some relatively arbitrary and (previously) hidden target.  That’s a big improvement. 

Another added plus:  Rasmussen has added a table providing monthly averages for the Bush job rating on his surveys dating back to February 2005.  That table will remain, with a monthly update, in addition to the usual 2-3 week window of 3-day rolling average results. 

For all the controversy over Rasmussen’s automated methodology, the biggest limitation of his free updates from my perspective has been that he only provides a 3-day rolling average.  The day-to-day variation usually looks like mostly sampling "noise" to me, yet the occasional random tick up or down inevitably leads to fruitless speculation somewhere in the blogosphere.  One suggestion for Scott Rasmussen:  Add a regularly updating last-7-day or last-30-day average, as those occasional ticks up or down in the 3-day results will look even more deceptively dramatic in comparison to the very stable 30-day averages. 

Finally, one quibble with Rasmussen’s characterization of the debate over party weighting:

One of the major methodological disputes among public opinion pollsters involves weighting data by Political Party. All agree that partisan affiliation is one of the best indicators of voting intentions and perceptions of the President. However, some firms and academic researchers believe that party affiliation changes on a regular basis. At Rasmussen Reports, we do not. We, along with many others, believe party affiliation is generally stable and that people switch their allegiance only rarely.

This view is supported by data and by common sense — how many people do you know that switch political parties on a regular basis?

To be fair, most that oppose weighting by party would also agree that "party affiliation is generally stable and that people switch their allegiance only rarely."  The issue is that a small percentage of Americans will switch their "allegiance" more easily, usually moving from a one party to the independent category and back again but rarely all the way from Democrat to Republican (or vice versa). I have written previously about evidence of this sort of movement, gathered by surveys that interview the same respondents more than once over a period of a few weeks or months.  Other research has shown evidence that the wording of the party identification question, the answer categories offered (see Franklin) or the questions that come before it may also alter the answer respondents give. 

Yes, when we aggregate large amounts of data over the course of a month or year, the day-to-day variation disappears and what trends do remain (such as those seen since early 2005) tend to be glacial (also documented exhaustively in this highly regarded book).  But some real yet fleeting variation may still occur on a day-to-day basis.  We have also seen some evidence of short term jolts (such at the aftermath of 9/11 or the 2004 Republican convention).  The problem is that when looking at individual surveys, we usually cannot tell how much of hte short term variation in party ID is real and how must just ordinary sampling "noise."  That’s really the nub of the debate, which will certainly continue.

But let’s give Rasmussen his due for moving toward greater disclosure and adopting a more sensible approach to party weighting.  That’s progress.