« The "Smoking Gun" Part II | Main | Koufax Award »

January 07, 2005

More Fun with Exit Polls

Salon.com's Farhad Manjoo has an excellent article, "More Fun With Exit Polls,"  out today on the continuing controversy that MP has been my focus since the election (though the article is part of Salon's "premium" content, you can get a free "day pass" by watching a brief advertisement). 

While Manjoo's story breaks no new ground that we have not already covered here, it is a very clearly written, well-balanced summary.   Those of you who simply cannot get enough of the exit poll issue will want to check it out. 

I may also be burying the lead:  Manjoo also quotes a certain blogger prominently.    So Mom, you'll definitely want to click here.   

Related Entries - Exit Polls

Posted by Mark Blumenthal on January 7, 2005 at 11:13 PM in Exit Polls | Permalink

Comments

Gday Mark - I havent really been following the exitpoll issue closely, but I read Farhad's article and he said "At midday Eastern time, NEP interviewers had spoken to about 8,000 voters" which kind of piqued my interest (it seems that he made a mistake - he seems to be referring to the 3.59pm poll, not midday). In any case, I took a closer look and noticed the discrepancy between the 11,027 and 13,660 numbers which you and others have also noted. However, I wonder if you were a bit dismissive when you said "The missing 2,633 interviews, presumably coming mostly from states in the Midwest and West".

The NEP issued a statement on Nov2 which appears to be their final summary at the end of the day about their activities for the day - it is still on http://www.exit-poll.net under the prominent link titled "National Exit Poll Methodolgy Statement" which links to http://www.exit-poll.net/election-night/MethodsStatementNationalFinal.pdf (note the word "final").

This document specifically says "The National exit poll was conducted at a sample of 250 polling places among 11,719 Election Day voters representative of the United States. In addition, 500 absentee and/or early voters in 13 states were interviewed in a pre-election telephone poll." That is, the NEP's apparently "final" statement explicitly states that they only did 12,219 interviews.

As we know, by 1pm the following day, the NEP was reporting that they had actually conducted 13,660 interviews - 12% more than they had officially stated - which is starting to test the limits of credulity, given that these people are supposed to count things for a living.

One hypothesis is that there was much interview-stuffing on Nov3 to produce the requisite swing to Mr Bush. To the extent that the NEP's 'final' statement is valid, then it would seem to render irrelevant all of the other discussions about chatty democrats and gender bias and the rest.

Separately, the consensus seems to be that the leaked numbers on Scoop are the legitimate NEP numbers - Mitofsky didn't claim otherwise in that delightful exchange with Mickey Kaus. However, some of the numbers seem curious - more here if you are interested http://wotisitgood4.blogspot.com/2005/01/exit-stage-left.html


Posted by: luke | Jan 9, 2005 4:21:26 PM

Luke, if I'm not mistaken, that NEP methods statement was circulated either with the first round of data, or prior to the election day. These were their targets based on historical turnout at sampled precincts; election turnout was higher than expected and therefore they had more samples than the statement said.

Also, they actually interviewed way more than 13,000 in that national poll. They probably interviewed about 2X that, but what is reported is only a sample of the actual polls. MP has dealt with these issues in previous posts, if I am not mistaken.

Posted by: Rick Brady | Jan 9, 2005 4:29:33 PM

Rick,
The lower numbers were targets? Really,
thats a little hard to believe because it doesnt make sense to report numbers that are targets to the nearest person..one would expect target numbers to be rounded..but who knows..NEP seems willing to do anything..I dont trust them one bit.

Posted by: Brian Dudley | Jan 9, 2005 6:22:47 PM

thanks rick and brian - as i mentioned, i havent been following it very closely - so i'm interested in what others have to say. but i do agree with brian - the statement is very specific - to the final person - and their language is in the past tense (and hasnt been updated)

i dont know anything about interviewing or exitpolls - but i did note that they had 1500 interviewers - which comes out to about 8 interviews per interviewer which seems like a low number - perhaps rick is correct in stating that there was 2X, altho that seems kinda odd.

Posted by: luke | Jan 9, 2005 7:44:58 PM

thanks rick and brian - as i mentioned, i havent been following it very closely - so i'm interested in what others have to say. but i do agree with brian - the statement is very specific - to the final person - and their language is in the past tense (and hasnt been updated)

i dont know anything about interviewing or exitpolls - but i did note that they had 1500 interviewers - which comes out to about 8 interviews per interviewer which seems like a low number - perhaps rick is correct in stating that there was 2X, altho that seems kinda odd.

i also posted at ruy's to see if anyone there had any further insight www.emergingdemocraticmajorityweblog.com/cgi/dr/mt-comments.cgi?entry_id=1011

Posted by: luke | Jan 9, 2005 7:46:55 PM

grrrr - i dont even know how to post properly. neither a pollster nor poster be.

Posted by: luke | Jan 9, 2005 8:10:59 PM

Luke and Brian, first what I know...

Read Mitofsky and Edelman's chapter in Presidential Polls and the News Media (1995) about the 1992 exits.

"Including the state polls, interviews were conducted with 177,000 voters on election day in 1,310 precincts....There was usually only one interviewer working at a polling place. Interviewers arrived before voting began and worked 50 minutes of each hour. During their 10-minute break, they did a hand tally of the number of respondents who said they voted for each candidate. Interviewers telephoned results of the hand tally and the individual responses to each question to a central site three times during the day. Before reading the individual responses, questionnaires were subsampled. The subsampling was controlled by the operator receiving the call at the central site. The subsample produced 62,024 questionnaires, of which 15,256 were from national questionnaires. The rest were from state exit polls" (pgs. 82-83).

I've communicated with Dan Merkle of ABC News who participated in the exit poll design. He confirmed to me that the subsample of the precinct interviews was followed again this year.

Now, what I don't "know" is whether the NEP methods statement was written before, during, or after the election. It was my assumption based on my understanding of how previous exit poll samples were drawn (based on historical turnout estimates) and the fact that anyone who goes to CNN's web-site knows that there were more interviews than were reported in the methods statement.

Posted by: Rick Brady | Jan 9, 2005 11:03:26 PM

Isn't the subsampling a way to save time on the phone on election day by not collecting the full voter profile, but still collecting voting data from the FULL sample?

In other words the subsampling only affects the voter profile data, but the vote tallies are from the full set of interviews conducted. Just want to make sure we all understand that the subsampling does not affect the predictions of Kerry/Bush %'s. Unless of course I've misunderstood Mark's post on this matter.

Posted by: Alex in Los Angeles | Jan 10, 2005 2:01:06 AM

thanks rick - although im less concerned about their methodolgy per se, and more interested in the fact that the NEP said that they conducted 12,219 interviews (however that is defined) and then the following day they apparently reported that they had conducted 12% more than that.

i dont know when the method statement was written either - but as Brian said, the level of specificity - to the final person - strongly suggests that it was written after the fact. Given that it was these last interviews which swung the result so dramaticallly, then there appears to be some serious unanswered questions.

If NEP's statement that it only conducted 12,212 interviews is true, then they apparently/presumably lied and simply created the final 1448 interviews in the office on Nov3 in order to get the number they wanted. In my book, that is called fraud.

Posted by: luke | Jan 10, 2005 4:50:11 AM

Alex,

If you look at the following .jpg of one of those "leaked" exit poll data (http://photos1.blogger.com/img/150/1911/640/11-2_7-30pm.jpg), you'll notice that the crosstab for the whole proportion is based on the subsample.

I do not know if the NEP members saw different data than the subscribers (the graphic is reportedly leaked from a subscriber), or if the projected calls were made on the full tallies that were phoned in. My guess is that the projections were made based on the full tally (Why collect the data).

From MP's early post on projections, I understand that projections of winners was hardly a factor here (i.e., battleground states weren't "called" for a long long time).

Alex, you wrote:
"In other words the subsampling only affects the voter profile data,..."

The data that everyone (Freeman, Simon/Baiman) has been "analyzing" (I say that loosely) is based on the subsample. The data Edison/Mitofsky are analyzing include all the interviews per precinct.
This is one more reason why I've been saying (on my blog) that the data are too "fuzzy" for rigorous statistical analysis where probability calcs like 662,000:1 are dubious. These probability calcs are highly sensitive to slight adjustments to the data.

Luke, something else to consider. I'm pretty sure that ALL the state sample sizes were off as well (although not all were under-counted). E.g, in CA the statement said they interviewed 2,105, but the CNN web-site shows 2,390 interviews. For the Alaska poll, they showed 1,194 interviews, but the CNN website shows 1,177.

These discrepancies are simply too obvious for the NEP to be trying to pull a fast one on the public. That is why I continue to maintain that these were circulated with the election data. However, I'll ask around...

Posted by: Rick Brady | Jan 10, 2005 9:07:46 AM

Cheers Rick - i've only been looking at the 'leaked' Scoop data, which is by region. CBS is using the same 13,660 number, and they seem to have pulled their state data http://election.cbsnews.com/election2004/state/state_us.shtml currently returns a 'Not Found'

I agree with you that it seems unlikely that they have pulled a fast one - yet there are obviously some outstanding questions with the data.

Interestingly, the % increases from the 7.33pm numbers to the Nov3 numbers are weighted toward the East and MidWest
East - 39%
MW - 31%
South - 13%
West - 20%

Again, I havent been following this issue closely - so i'm playing catchup and apologise if i ask questions that have been asked/resolved a thousand times. I dont know the slightest thing about polling - my experience is in industries where 12% mistakes are significant...

On a separate note - Farhad in Salon refers to midday polls - was that simply a factual error?

Posted by: luke | Jan 10, 2005 10:35:09 AM

the nashua advocate points out other discrepancies in the size of the pool (disclaimer: i sent them an email)
http://nashuaadvocate.blogspot.com/2005/01/news-election-2004-nep-adds-hundreds.html

They talk about 'hundreds' of additions, which is fine - but the reality is that its the percentages which are significant here - we are talking about a 12% discrepancy in the number of interviews, in an environment where 3% is apparently considered a mandate... and if the apparent swing to mr bush was already near impossible in the final sample of (apparent) interviews - it is definitely impossible if there was actually only 12,212 interviews.

theres also something odd in the fact that if you look at the difference between the 7.33pm numbers and the numbers from Nov3, most of the % gain is from the East - how can it be that the East gained 31%, and the West gained only 17% from that point in time - given the time difference? bad cell phone coverage?

Posted by: luke | Jan 10, 2005 11:13:04 AM

Cell phone coverage? How would that play a role in exit polls?

Rick:

Thank you for explaining the subsampling found in Freeman's data. I think that is actually a revelation, at least to me.

So, MOE's are based on the subsample number of interviews? Would working with the full sample improve the MOE substantially?

This subsampling issue seems to add another significant layer to the matter of figuring out what went wrong with the exit polls. Or am I not understanding the implications correctly?

Thanks for any further clarification you or Mark can provide.

Posted by: Alex in Los Angeles | Jan 10, 2005 12:28:23 PM

First things first. I got word from Jennifer Agiesta of Edison Media Research. She has been VERY kind to me over the past month in providing me with information. Re the discrepancies:

Rick,

The CNN web site is displaying the number of unweighted cases that are being used in the crosstab for the presidential vote.

The methodology statement includes the actual number of respondents who filled out the questionnaire.

These two numbers can differ for two technical reasons:

The first reason is that some respondents fill out the questionnaire but skip the presidential vote question. For example in Alabama the CNN site shows 736 respondents. The methodology statement show 740 respondents. This is because 4 respondents chose not to answer the question on how they voted for president and are not included in those crosstabs but they are still included in the data file because they may have filled out how they voted in other races that day such as the Senate race.

The second reason is that respondents from the national absentee/early voter telephone survey received all four versions of the national questionnaire while election day respondents only received one version of the national questionnaire. Thus, these respondents are included 4 times in the unweighted data (once for each version of the questionnaire) but their survey weights are adjusted down so that in the weighted data each national absentee/early voter telephone survey respondent only represents one person.

Again the methodology statements state the correct number of total respondents interviewed.

I hope that this explains the differences in how these numbers were reported.

Jennifer

------

There you have it. The telephone surveys were quadruple counted in the sample sizes reported on the CNN website. Now, the question to me seems to be: Why not weight these surveys, before sending them around to their clients? Doesn't make sense to me, but I don't think this is some conspiracy here.

Luke, as far as the regional questions go, I think MP might be the best one to answer this (and he may be trying to do so now). I have ordered the 2000 exit poll data from the Roper Center for some analysis and hopefully the data sets will include some explanation of how the geos were defined. Although, I'm not too hopeful there will be much of use here.

Alex, we don't know much about cell phone coverage error. I've read articles that suggest its a big issue and other articles that suggest its not an issue. Also, we don't know what rate of "misses" or "refusals" occurred with the phone calls, let alone cell phone calls. Perhaps MP has already, or would be willing to comment on the cell phone question. I think this is an open debate in the polling world right now.

Also Alex, re the subsampling procedure. I see three sampling stages here: 1) selection of a sample of precincts (cluster); 2) sample of voters; and 3) sample of the sample of the voters.

The way I see it, the exit poll data circulated on election day and made available to subscriber clients (the leaked Scoop PDFs) were based on a sample of a sample of a cluster sample.

What does this do to the MoE? Well, nothing to the MoE if you consider the table that WAS circulated with the data on election day to subscribers (I've confirmed this much). If we look at those tables, there is a range of sample sizes and the clients were supposed to look at the sample size and determine the corresponding MoE. This a VERY rough guide, but according to Mitofsky (e-mail to me), it was developed based on a calculation of the design effect according to the poll design (Merkle and Edelman participated in this calculation).

Therefore, the data Edison/Mitofsky are analyzing as we speak is the full set of data. In fact, the data that Edison/Mitofsky may have used to "project" winners on election night, may have been the full tabs "corrected" for non-responses . (They "corrected" for non-responses in the 2004 primaries anyhow - the question is - what if what you "know" about the non-responses because of age, sex, and race is not "true"?).

------

Bottom line the way I see it. The data and methods are too fuzzy to make any hard statistical analysis. That means, when folks like Freeman and Simon/Baiman throw around VERY precise calculations of odds and then base conclusions on those odds, they either: 1) are trying to BS everyone; or 2) they really don't know what they are doing. Look; anyone who glances at the data knows it is skewed. It doesn't take a PhD...I mean...an excel spreadsheet and a 2000 analysis of the 1996 exit polls to figure that out.

The NEP holds the cards here. They have the data in the form that they need to analyze it correctly. These data will eventually be made public (via Roper Center).

Sure, some will say that these data are not trustworthy and may have been "cleansed" by the NEP (why? I don't know), but I won't subscribe to that theory. Edison/Mitofsky owes it to their clients to analyze the data before the likes of Freeman and Simon/Baiman write severely flawed reports.

I think their distinguished careers have earned them at least this courtesy.

Posted by: Rick Brady | Jan 10, 2005 3:30:11 PM


One point that should be made for the benefit of foreign readers and those who were asleep during their Civics class is that the U.S. Electoral College system can forgive more sins than the Pope.

There is nothing about "democracy" in it, and so there is no possibility of tainted results propagating past the electoral college. All the math required to tabulate it can be doubled checked by anyone with a second grade education.

In the last analysis, democracy in America depends not on exit polls or even the popular vote, but on the willingness of alleged democratic states to certify their electoral votes, and the willingness of Congress to seat them. Disputes are not resolved by exit polls or re-votes or recalls or even courts, but in the last analysis by Congress.

In order to have a "failed election" similar to the one alleged in the Ukraine, under the design of the U.S. Constitution, one would have to subvert 26 out of 50 democracies--and if successful, need I add the country would already be in far greater trouble than

Perhaps sites like this would do a public service if, among the endless talk about exit polls and voting, they pointed out that the political design of the U.S. is precisely to allow inherently uncertain processes to gain net certainty, in defiance of statistics, information theory, and electoral democracy--by incorporating just a few people and some things called "Constitutions" and "Laws" in the aggregation stage. This takes out the mathematics--even to the extent of legitimating fraud--on *purpose*.

In summary: the outcome of U.S. national elections is certain and legitimate after the legal and congressional process has run its course, almost no matter *what* happens.
The legitimacy has *nothing* to do with democratic voting or elections, by design, and by excellent design--and for precisely forseeing the reasons that we cannot agree on here, namely that if we take people out of the process, and put in mathematics or machines, we no longer have a political process but a noise generator.

That means: if you feel the democratic process has been subverted, your task is to reform all 50 *states* or whichever ones you don't like. If Ohio is not a democracy, that is a problem for Ohioans. We, as a nation, have a political system that can survive carpetbagging in all the former states of the Confederacy.

Remember this the next time someone wants you to overthow the U.S. electoral system. We are not a democracy, at the national level, so we can be ruled democratically.

Posted by: John Goodwin | Jan 10, 2005 3:50:36 PM

Hi Mark

I still haven't gotten any responce to the question I posted a couple of days ago. But it goes a long with this topic to so I'll point it out again. Can you explain this

http://www.cnn.com/ELECTION/2004/pages/results/states/US/P/00/epolls.0.html

The smoking gun is the final exit polls that CNN and others have up at this time.

These are the numbers (13,660) put up to dispel the "problems" with the early numbers and the ones they have used to tell us what we think. They are OBVIOUSLY misleading/wrong (see web page above and below). The PROOF is about 2/3's maybe 3/4's of the way down, the question is who did you vote for in 2000. The problem is obvious 37% Gore 43% Bush in 2000 that is WRONG. Again the people that did the polling know that's not right, the people that posted it know it's not right. They didn't do any fancy weighting to adjust for that, as I’ll show below. It's that simple. It's more obvious than the type setting on the font of an old document. Anyone with a statistical background would know it is not correct. The only thing I wonder about is how that data got in the stats at all. Maybe someone on the inside is trying to say something?? I don’t know?

Do you understand how weighted averages work?? The sum of the fraction times the amount

Numbers from the data (see again see web poll data below go to the question who did you vote for in 2000). The 51/49% Bush win is what we are being told. If you plug in the straight numbers the data comes out as they what you to believe. This works with other number too so that shows no other factor are included:

.17*45+.37*10+.43*91+.03*21=51.11% Bush

.17*54+.37*90+.43*9+.03*71=48.48% Kerry

With equal amounts of Gore and Bush AS WE KNOW?? IT WAS.

.17*54+.40*90+.40*9+.03*71=50.91% Kerry

.17*45+.40*10+.40*91+.03*21=48.68% Bush

Again, any one who puts out a poll that doesn’t have equal amounts of Gore to Bush has to know the data collection is faulty, and that any information gleamed from it is also wrong. Also if 17% of the people that didn't vote last time went 54% Kerry and 45% Bush and that is in a Bush favoring sample. The switch Bush to Kerry and Gore to Bush are about equal, looking at that there is really no way to say that Bush won.
But that is not my point the point is that CNN and others are putting up data that is obviously wrong! Why is not really important to the point. How is not really either. The point is the data is wrong and it is still up. Any explaination?

Posted by: Rolf Unterleitner | Jan 10, 2005 4:11:46 PM

Hi Mark

I still haven't gotten any responce to the question I posted a couple of days ago. But it goes a long with this topic to so I'll point it out again. Can you explain this

http://www.cnn.com/ELECTION/2004/pages/results/states/US/P/00/epolls.0.html

The smoking gun is the final exit polls that CNN and others have up at this time.

These are the numbers (13,660) put up to dispel the "problems" with the early numbers and the ones they have used to tell us what we think. They are OBVIOUSLY misleading/wrong (see web page above and below). The PROOF is about 2/3's maybe 3/4's of the way down, the question is who did you vote for in 2000. The problem is obvious 37% Gore 43% Bush in 2000 that is WRONG. Again the people that did the polling know that's not right, the people that posted it know it's not right. They didn't do any fancy weighting to adjust for that, as I’ll show below. It's that simple. It's more obvious than the type setting on the font of an old document. Anyone with a statistical background would know it is not correct. The only thing I wonder about is how that data got in the stats at all. Maybe someone on the inside is trying to say something?? I don’t know?

Do you understand how weighted averages work?? The sum of the fraction times the amount

Numbers from the data (see again see web poll data below go to the question who did you vote for in 2000). The 51/49% Bush win is what we are being told. If you plug in the straight numbers the data comes out as they what you to believe. This works with other number too so that shows no other factor are included:

.17*45+.37*10+.43*91+.03*21=51.11% Bush

.17*54+.37*90+.43*9+.03*71=48.48% Kerry

With equal amounts of Gore and Bush AS WE KNOW?? IT WAS.

.17*54+.40*90+.40*9+.03*71=50.91% Kerry

.17*45+.40*10+.40*91+.03*21=48.68% Bush

Again, any one who puts out a poll that doesn’t have equal amounts of Gore to Bush has to know the data collection is faulty, and that any information gleamed from it is also wrong. Also if 17% of the people that didn't vote last time went 54% Kerry and 45% Bush and that is in a Bush favoring sample. The switch Bush to Kerry and Gore to Bush are about equal, looking at that there is really no way to say that Bush won.
But that is not my point the point is that CNN and others are putting up data that is obviously wrong! Why is not really important to the point. How is not really either. The point is the data is wrong and it is still up. Any explaination?

Posted by: Rolf Unterleitner | Jan 10, 2005 4:11:46 PM

Rick,

Hi! This is the News Editor of the Nashua Advocate. I've been following the discussion above with great interest, and the main problem I have with what I've heard -- despite not being one, generally speaking, for conspiracy theories -- is that we already *know* Mitofsky/Edison Media will artificially add and/or subtract "phantom voters" to their published sample size.

How do we know this?

Because they did it with the Ohio state-level exit poll, and have *admitted* to doing it.

Recall: the third of the three Election Day exit polls from Ohio -- valid until the early morning of November 3rd, 2004 -- showed 1,963 respondents. Mitofsky admits that the data was then "adjusted" to reflect raw vote totals. How was that done? By *subtracting* 31 Kerry "voters" and *adding* 88 Bush "voters" (the new poll had 2,020 voters as its sample size, but the new percentages for each candidate showed that no "new" voters had been queried -- the existing tallies had simply been manipulated or, as Mitofsky says in his infuriating Orwell-speak, "adjusted").

Now, what does Mitofsky's willingness to add "phantom voters" to his sample size do to public confidence in his exit-polling?

Right now, The Nashua Advocate is reporting *four separate* allegedly "final" sample sizes for the National Exit Poll of November 2nd. How in the world is the public to distinguish between the justification Edison Media gave you for this discrepancy -- which indeed sounds entirely innocent -- and the entirely non-innocent fact that, in the Ohio poll, the sample size was changed using "phantom voters" who didn't really exist and had never been queried as to their preferences?

So, when the N.E.P. releases its data in the coming weeks, how do we know if it was "adjusted" or not? And given that Mitofsky -- in an e-mail to MSNBC reported on by Keith Olbermann -- has expressed quite clearly that he thinks he can claim his polls are "accurate" based on *adjusted totals*, why should I or anyone else feel confidence that Mitofsky understands the difference to the American people between releasing adjusted and non-adjusted poll data?

What's to stop him from releasing adjusted data and saying this is the "right" data, and the public has no right to see any other data because it isn't the "right" (read: tautologically correct, by filtering in "real-world" vote tallies) data?

Thanks for any insights you can provide.

-- The News Editor

Posted by: The Nashua Advocate | Jan 10, 2005 4:40:51 PM

Rolf,
It is a well known fact that when polled after an election people lie about who they voted for. (Dont ask me why...its absolutely bizarre and a cowardly thing to do...but apparantly they do) A few people who voted for the loser will say they voted for the winner...people just like winners. Or maybe people who didnt vote...lie a say they voted for the winner.

Posted by: Brian Dudley | Jan 10, 2005 5:08:34 PM

More here... real quick... then back to feeding the kids...

More on the subject of exit poll methodology from Merkle and Edelman (2002), “Nonresponse in Exit Polls: A Comprehensive Analysis” from Survey Nonresponse, Eds Groves, et. al. Pp 243-257.

“The VNS exit polls are conducted using a two-stage sampling design. In the first stage, a stratified, systematic sample of precincts is selected in each state, proportionate to the number of votes cast in a previous election. In the second stage, interviewers systematically select voters exiting polling places on Election Day, using a sampling interval based on the expected turnout at that precinct. The interval is computed so that approcimately 100 interviews are completed in each precinct…

“Interviewers are instructed to work with a polling place official to deermine the best place to stand. Ideally, the interviewers are located inside the polling place where all voters must pass them. Unfortunately, sometimes interviewers must stand outside, some distance from the door…

“Interviewers take a 10-minute break from interviewing each hour to tally the responses to the vote questions and their observations of the nonrespondents. Interviewers call VNS three times during the day to report their data. In local time, the first call is around 9:00AM, the second around 3:00 PM, and the last call shortly before poll closing. During each call, interviewers report their vote and nonresponse tallies and read in the question-by-question responses from a subsample of the questionnaires…”

John - others think you are an enigma (and sometimes I do as well), but welcome back! It's been a while.

Nashua Advocate News Editor, if you could provide the sources and links to the quotes from Mitofsky, I'd like to check it out.

BTW - I HIGHLY doubt that Edison/Mitofsky will try to rely on sampling error to explain the discrepancy. They will partially fall on the sword (training, etc) and partially carve it up to differential non-response.

Remember, there are about 77,000 additional surveys out there that they have access to and we do not. Who knows how this will affect the comparative analysis?

"Adjusting" the data to ensure a valid sample is standard stats and is REQUIRED. The subject of weighting is often where pollsters and the public often go awry. Some weighting is absolutely necessary and routine, while other types of weighting involves "educated guesses" (but is also routine).

For example, if they weighted the non-responses according to what they "know" about the demographics (they can guess the age, sex, and race of the nonrespondent), then this is an example of an "educated guess." They take what they "know" about these demos and they adjust their sample accordingly. If for some reason what they "know" is uniformly different from what "is true", then the poll will have bias.

The Nashua Advocate News Editor and Rolf, I have a strange feeling that MP will have more on these subjects... I've shared about all I know.

Posted by: Rick Brady | Jan 10, 2005 5:08:41 PM

Brian Dudley, when I hear things like "It is a well known fact" without some type of source, I start to twitch. Especially, if I don't know that fact. :-)

Posted by: Rick Brady | Jan 10, 2005 5:10:19 PM

Nashua editor, Rick:

Do I understand things correctly, Nashua seems to describe Mitofsky adjusting the SUBSAMPLE from 1963 to 2020?

If it is the subsample that is being adjusted, that brings up two points:
1. So, is the adjustment Nashua points out what we call weighting to match the election results? So the reweighting involves adjustments to the subsample? That doesn't sound like reweighting as I understand it. Or is this an additional step separate from the reweighting to match the election results? Boy this is started to get conceptually complicated on top of the statistics. Help.
2. Why wouldn't Mitofsky incorporate his full data sample and disgard the subsample as soon as possible? It looks like his subscribers are still unnecessarily limited to his subsample based data.

Please note that if my questions indicate my misunderstanding of the issues, I apologize.

Posted by: Alex in Los Angeles | Jan 10, 2005 5:44:10 PM

Alex, that appears to be the claim. The full sample would be more like 4,000.

re: #1. I don't know anything more than what was described to me by the NEP. Without more information (like sources of the claims) from Nashua, who knows?

#2. I suspect that by the time the NEP had the full dataset (whenever the interviewers faxed or mailed them in), the election was over and they automatically switch into the "understanding the outcome" mode (i.e., predictions are over, now let's weight to election result and see what it tells us).

Who knows...Maybe historically the differences between the full sample and the subsample aren't statistically significant? Therefore, why bother?

Posted by: Rick Brady | Jan 10, 2005 6:17:02 PM

Alex - my question re the cell phone coverage was supposed to be part tongue-in-cheek - i thought that the excuse may have have been that 'we couldn't call the results through'

here are the interview numbers (the times are Eastern, apologies for the formatting):
4pm 733pm 3-Nov
East 1746 2077 2888
West 1747 2203 2640
MW 2069 2804 3676
South 2787 3943 4456

Somehow, the East gained 39% more interviews *after* 7.30pm, and the West only gained another 20%, despite the time difference.

Posted by: luke | Jan 10, 2005 7:27:57 PM

Luke, hang in there... I think I hear MP typing...

Posted by: Rick Brady | Jan 10, 2005 8:05:29 PM

The comments to this entry are closed.