Unfortunately, I’m at least a week late with this recommendation, but perhaps for true political polling junkies it will be better late than never. For MP-like commentary on political polling in the United Kingdom, no one does it better than Anthony Wells’ UK Polling Report (actually, it might be more appropriate to describe MP as "Wells-like," since he was blogging long before I started). His site now includes an impressive archive of British political polling. His review of the performance of the polls in last week’s parliamentary election is noteworthy especially since their accuracy came as something of a surprise.
Here is Well’s post-election take on the UK pre-election polls:
So, with pretty much everything except Harlow counted, how well did
the pollsters do? The bottom line is that everyone got it right –
trebles all round! While NOP take the prize, having got the result
exactly spot on, not only did all the pollsters get within the 3%
margin of error, they all got every party’s share of the vote to within
2%. Basically, it was a triumph for the pollsters.RESULT – CON 33.2%, LAB 36.2%, LD 22.7%
[Update 5/10: These results were posted on Wells’ blog on 5/7. As of today, BBC is reporting Conservative 32.3%, Labor 35.2%, Liberal Democrat 22.0%. A am leaving Well’s error estimates in place, although the reader should note they need to be recalculated]
NOP/Independent – CON 33%(-0.2), LAB 36%(-0.2), LD 23%(+0.3). Av. Error – 0.2%
MORI/Standard – CON 33%(-0.2), LAB 38%(+1.8%), LD 23%(+0.3). Av. Error – 0.8%
Harris – CON 33%(-0.2), LAB 38%(+1.8), LD 22%(-0.7). Av. Error – 0.9%
BES – CON 32.6%(-0.6), LAB 35%(-1.2), LD 23.5%(+0.8%). Av. Error – 0.9%
YouGov/Telegraph – CON 32%(-1.2), LAB 37%(+0.8), LD 24%(+1.3). Av.Error – 1.1%
ICM/Guardian – CON 32%(-1.2), LAB 38%(+1.8), LD 22%(-0.7). Av.Error – 1.2%
Populus/Times – CON 32%(-1.2), LAB 38%(+1.8), LD 21%(-1.7). Av. Error – 1.6%The
other two pollsters, Communicate Research and BPIX, conducted their
final polls too early to be counted as proper eve-of-poll predictions,
but, for the record, both their final polls were also within the
standard 3% margin of error. Their average errors were 0.9% for BPIX
and 2.1% for Communicate.The British Polling Council
have a press release out with the same information (although they
include the “others” in the average, and use rounded figures for the
results, hence the slightly different figures. It doesn’t change the
result – everyone was right and NOP did best).
On Election Day, Wells posted the results of the national exit polls, which did even better:
MORI/NOP’s exit poll shows a share of the vote of CON 33%, LAB 37%, LD 22%. It predicts 44 Conservative gains and 2 Liberal Democrat gains for a Labour majority of 66.
Of course, the actual share of the vote was Conservative 33.2%, Labor 36.2% and Liberal Democrat 22.0%, translating into a 67 seat Labor majority.
[Update: Again, these results were posted on Wells’ blog on 5/7.
As of today (5/10), BBC is reporting Conservative 32.3%, Labor 35.2%, Liberal
Democrat 22.0%. I have changed the error computations below to reflect the updated results. Also a reader emails to advise that the final Labor majority will end up being
66 seats not the current 67 seats after a special by-election to be held soon to replace a recently deceased LD MP. The
district should go Conservative in the by-election so the Labor majority will be
66 seats within a few weeks exactly matching the projection from the exit
polls].
So does the UK experience show, as some would seem to believe, that exit polls are flawless elsewhere but troubled only in the US? Hardly. In fact, much of Well’s enthusiasm comes from the previously problematic performance of the UK pre-election and exit polls. Let’s start with the exit polls. The error on the margin in last week’s MORI/NOP exit poll was only 1.10 percentage point in Labor’s favor. As Well reports in a summary, the NOP/BBC exit polls showed a much greater skew to Labor in 2001 (2.7) and especially in 1997 (5.2) and 1992 (3.6).
What about the pre-election polls? The average error for each candidate estimate last week across all the polls reported by Wells (I did the math) was 1.10 percentage point. The comparable average error was higher in 2001 (1.7), 1997 (2.5) and 1992 (3.3). The error on the margin has shown a consistent skew to the Labor party, overestimating its lead over the Conservatives in 2001 (3.5), 1997 (4.0) and especially 1992 (8.3). The slight error in Labor’s favor this time (1.76) is obviously small by comparison.
What is interesting about the consistent skew to Labor in recent polling is that pollsters have named it: "Shy Tories" or the "Spiral of Silence" effect (the latter based on the book of the same name by Elisabeth Noelle-Neumann). As defined by the BBC guide to polling methodology, it means "people who do not like to admit they support a certain party but who vote for them nonetheless." Some UK pollsters now reallocate undecideds to counter this phenomenon, although Wells’ wrap-up notes that "the spiral of silence adjustment made Populus’s final poll less accurate."
It is also interesting that there may be remnant traces of the Labor skew in this year’s result. Five of seven prelections polls and the exit poll overestimated the Labor vote by one percentage point or more (though my favorite binomial calculator tells me there is a 23% probability of tossing a coin seven times and having it come up heads five times). Wells — who identifies himself as a supporter of the Conservative Party — concludes that the "lingering bias" is "small it is hardly worth worrying about."
One characteristic that the UK polls shared with their US counterparts is the way widely varying results tended to converge in the final week. More from Wells:
What is interesting is the comparison between the final result and the
polls during the campaign – the results from YouGov during the campaign
were pretty close to the final result throughout, especially after the
first few polls that showed the parties neck and neck. In contrast
during the campaign the phone pollsters showed some whopping great
Labour leads that disappeared in their final polls – of all the phone
polls during the campaign, only one (MORI/Observer, published on the
1st May), did not report a Labour lead larger than the 3% they finally
acheived. Of YouGov’s last 10 polls of the campaign, 8 showed a Labour
lead of 3 or 4 percent. It doesn’t, of course, necessarily mean that
YouGov were right – the “real” Labour lead at that time could have been
larger, only to be reduced by a late swing to the Lib Dems – hence the
reason why we only compare the eve-of-poll predictions to the final
result.[Note: YouGov draws samples from a panel of pre-recruited respondents and conducts interviews online]
Finally, back to exit polls: It is noteworthy that the UK had no controversy about "leaks" of "early" exit polls and no dark suspicions about the re-weighting of the exit polls to match the actual count because neither phenomena occurred. The British exit pollsters release their "final" results as the polls close for all to see. It helps, of course, that Britain has a uniform poll closing time, but survey research in the UK seems to have survived the release of imperfect results. In fact, it is possible that this year’s improvements resulted from the public disclosure of those previously problematic surveys. Hopefully Wells will devote some future posts to discussing what, if anything, the UK exit pollsters did differently this time.
Also consider that the previous problems with the UK exit polls did not produce election fraud conspiracy theories. Why not? One reason, as our friend Elizabeth Liddle points out, is that the count in the UK is "utterly transparent." Paper ballots are sorted and counted in public at a centrally located place, open to all who wish to observe. People have faith in the result because of this transparency. Another reason, as Liddle puts it, "why auditable elections are so important."
Hear, hear!
5/10 – Update: Anthony Wells emails with a postscript on the real reason why exit poll leaks are so rare in the UK. They are illegal. Leakers are subject to a fine of up to £5,000 or 6 months in prison. Here is the text of the law:
No person shall, in the case of an election to which this section
applies,
publish before the poll is closed—(a) any statement relating to the
way in which voters have voted at the
election where that statement is (or
might reasonably be taken to be)
based on information given by voters after
they have voted, or(b) any forecast as to the result of the election
which is (or might
reasonably be taken to be) based on information so
given.
Why no such law in the US? Well, there’s that funny thing called the First Amendment…
MP wrote:
“Also consider that the previous problems with the UK exit polls did not produce election fraud conspiracy theories. Why not? One reason, as our friend Elizabeth Liddle points out, is that the count in the UK is “utterly transparent.” Paper ballots are sorted and counted in public at a centrally located place, open to all who wish to observe. People have faith in the result because of this transparency. Another reason, as Liddle puts it, “why auditable elections are so important.”
Hear, hear!”
So, now, we finally get to it. Auditable elections. Thank you Mark and Febble.
Febble also called exit poll discrepancies a possible red herring. I would agree in that in America the real problem is that our election counts have no way of being verified to the satisfaction of serious minded folks.
In a saner political culture would we really accept the nonsense of UNVERIFIABLE elections? Aren’t the stakes too high to trust closed door and black box vote counting?
I will say this, is anyone else thinking it would be a good idea to remove all election officials that continue to insist we should move to e-vote, no paper trail, and generally unverifiable vote counting systems? I would vote for Connie McCormick of Los Angeles as the first to go. Any other nominations?
What is interesting is that the exit poll numbers in Great Britain show the same left-wing “skew” they’ve show in the U.S. Does anyone have numbers for Australian, Canadian, French or German election exit polling to see if this trend continues?
Mark, if leaks of biased exit poll results lead to a bandwagon effect (or any effect at all), then I’d say the 1st amendment right to speech takes a backseat to ensuring free, fair, and unmolested elections. It doesn’t take much to “throw” a poll and then leak the results…
I’m no conspiracy theorist, but imagine George Soros encouraging his MoveOn followers to flood the NEP with applications to be exit pollsters, having them forge 1 in 15 or so interviews (giving only a slight bias), then buying someone with access to the data to ensure a midday leak of results he knows are biased. The same could be said about Karl Rove since he has unlimited access to the Halliburton treasury 😉 Heck, it’s certainly easier than rigging an election.
I think I’m in agreement with the UK law.
Mark writes:
“So does the UK experience show, as some would seem to believe, that exit polls are flawless elsewhere but troubled only in the US? Hardly”
Please! The Brits had 645 winner-takes-all elections to predict and got them all perfect. The greatest pollsters in the greatest nation on earth got it all “wrong”, and this is “hardly” significant?
Sjerp:
I don’t think the exit polls got all 645 results perfect. Many of these are “safe” seats”. The polls got the share of the vote right, which is impressive, and this would have told them the likely “swing” from which they can compute how many marginal seats are likely to fall, and to whom. They got this roughly right too, but I don’t think they necessarily picked exactly which ones would fall.
On election night, the TV stations usually list the constituencies in rank order of how “safe” they are for any given party, and as information comes in about the “swing” – from the exit polls, at first, then the actual results – they compute how far down the list the seats should “fall”. But in practice the ones that fall are not the ones predicted. Putney, once my constituency, went to the Tories on a big swing. Extrapolated this would have meant a hung parliament. However, it turned out to be exceptional. I doubt the exits got that one right, but I don’t know.
What I do know is that in 1992, the exits predicted a hung parliament with a Labour majority – very exciting after 11 years of Tory rule. But John Major won, with an absolute majority of 23 seats. He was a a much-mocked leader, with an Iraq war on his hands, presiding over an economic downturn. The explanation given for the exit poll error was that people had voted out of fear of something worse (Labour) but had been ashamed of the fact. No proof, but “shy Tories” were certainly factored in to subsequent polls.
Funny how, on November 3rd last year, I had that strange sense of deja vu….
Interestingly a couple of years later, a poll that asked people how they had voted in the previous election gave Labour a thumping retrospective majority.
Mark,
Just ot let you know, there is a good reason why the BBC’s shares of the vote don’t match mine. Opinion Polls in the UK almost always cover only Great Britain (i.e. they don’t include Northern Ireland).
In contrast, the BBC’s shares of the vote are for the whole of the UK, so they do include Northern Ireland. For comparisons with the opinion polls though, you need to work out shares of the vote for just Great Britain. Hence the difference.
Anthony
Oh, and like Febble says, the exit poll didn’t actually get all 646 seats right. For example, the exit poll predicted 8 Conservative gains from the Liberals Democrats (the 3rd party in the UK). In reality there were only 5 such gains. It’s just that their various errors cancelled each other out so they ended up getting the majority right.
Minor correction, Mark:
“Also a reader emails to advise that the final Labor majority will end up being 66 seats not the current 67 seats after a special by-election to be held soon to replace a recently deceased LD MP. The district should go Conservative in the by-election so the Labor majority will be 66 seats within a few weeks exactly matching the projection from the exit polls”
The “by-election” (strictly speaking, the delayed holding of the general election in that constituency) is in a seat with a Conservative incumbent. The LD candidate who died was not the incumbent, but under British law this still requires the postponement of the poll.