A Tricky Election!

By Adam Drummond – head of political polling at Opinium

Like all other polling companies we have spent the weeks since the general election analysing our data to try and determine why our final numbers failed to accurately predict the result. Our figure for the Liberal Democrats was spot on while those for UKIP and the SNP were within one percentage point of the result although given the geographic focus of the SNP’s support this error was proportionately larger in Scotland. The main areas where we made errors were understating Conservative support and overstating support for Labour and the Greens.

As a reminder of how the pollsters fared, the table below shows the final results of each member of the British Polling Council before the election.

 

Method

Tory

Labour

Lib Dem

UKIP

Green

Other

AVERAGE ERROR

GE2015 Result (GB)

 

37.80%

31.20%

8.10%

12.90%

3.80%

6.30%

-

ComRes

Telephone

35%

34%

9%

12%

4%

6%

1.52%

Opinium

Online

35%

34%

8%

12%

6%

5%

1.76%

Ipsos MORI

Telephone

36%

35%

8%

11%

5%

5%

1.76%

YouGov

Online

34%

34%

10%

12%

4%

6%

1.92%

ICM

Telephone

34%

35%

9%

11%

4%

7%

2.12%

Populus

Online

33%

33%

10%

14%

5%

6%

2.16%

Panelbase

Online

31%

33%

8%

16%

5%

7%

2.60%

With that in mind, this post will take you through the results of our “exit poll” (in reality a re-contact survey of respondents who voted) and some possible explanations for what went wrong and how we can fix it.

Key lessons learnt from our review:

There is no evidence of a late swing from our data which means the issue is who we asked and how we counted themWhether because of social desirability bias, or because polls are generally answered by more politically enthusiastic people, all polls over-exaggerate likely turnout which leads to over-representation of some low-turnout groups who typically favour LabourWhen we weight the sample of our final poll to match the demographics of the 2010 voting population – rather than the adult population as a whole - we get much closer to the actual resultThis suggests that political polls will need to take greater account of differential turnout between demographic groups in future but this itself presents risksFurther issues specific to Opinium include our understating of the SNP, which most likely contributed to our overstating of Labour, as well as underrepresenting respondents aged 65+ who tend to be the most pro-Conservative

There is no evidence of a late swing – analysing our re-contact survey

On Election Day itself we sent a survey out to everybody who said they were likely to vote in a recent poll and asked them to fill it out only once they had voted.

Around four and a half thousand people answered this, only a fifth of the official exit poll but significant nonetheless, and if there had been a substantial late swing to the Conservatives then we should have seen some of it here:

Election day re-contact survey

Raw figures

Weighted to match final poll demographics of likely voters

With demographic and party propensity weighting

Conservative

33%

32%

33%

Labour

32%

32%

33%

UKIP

15%

15%

12%

Liberal Democrats

9%

8%

9%

SNP

5%

5%

4%

Green

5%

5%

6%

In order to make this representative we applied similar weights to make the profile, demographically and politically the same as that of the likely voters in our final pre-election poll. However all this gave us was the same dead heat or tiny Tory lead. The Labour figure was slightly more accurate but this again underestimated the Conservatives while overestimating UKIP and the Greens.

Ultimately the point of showing this information is that there is no evidence of a late swing to the Conservatives among our voting intention sample and therefore we cannot claim that our pre-election poll was largely correct but rendered inaccurate by a late swing.

The percentage of those who said they would vote Conservative who followed through on that was 91%, identical to the Labour figure while those for the Lib Dems, UKIP, SNP and Greens were 83%, 88%, 98% and 82% respectively. The people in our sample did not change their votes on the day itself in anything like sufficient numbers to explain the final result so we must instead look to who we asked and how we counted them.

Who answered our poll? Social desirability bias or a sample of political enthusiasts?

Our final pre-election survey was sent out to just under 3,000 people on Monday 4th May and was completed by the end of Tuesday 5th May before being published on Wednesday the 6th. It produced the figures in the table above and a fuller explanation of how they were compiled can be found later on in this post.

In that poll 81% said that they were likely to vote and told us which party they would vote for compared to an official turnout of 66%. Even restricting to those who “would definitely vote” gives us 76%, still significantly above the actual turnout.

In fact if we look at other polls we see similar results.

ICM’s final voting intention figures come from 1,544 respondents out of 2,023 equivalent to 76%In YouGov’s final poll of 10,307 people, 76% said that they were 10/10 certain to vote although it is unclear how many of these are included in their final voting intention figuresFor Ipsos MORI 873 out of 1,096 respondents (80%) say that they are absolutely certain to vote

In all cases further adjustments are made to the final figures but the point of bringing these numbers up is that people who respond to surveys, whether on the telephone or online, are more likely to say that they will vote than actual turnout implies. This is either an example of social desirability bias – something that online polls should in theory be less susceptible to given the lack of a human interviewer – or evidence that people who respond to polls generally are more politically engaged and enthusiastic than the population at large.

Either we are counting non-voters who themselves do not turnout on the day or we are counting too many people who are voting from groups that do not turnout in the numbers suggested by our polls.

In either case, polls exaggerate turnout but because different demographic groups turnout at different rates, this has a disproportionate influence on some groups and thus on final vote share.

So what is the effect of this?

The table below shows the different effects on each age group using turnout in the 2010 election as a guide. Data for 2015 is not yet available but, as the overall turnout in 2015 was very similar to 2010, we have worked on the assumption that turnout levels within each age group will be similar as well and have seen no evidence yet to disprove this.

Age group

Implied turnout in our poll

2015 turnout in age group (source)

Implied turnout as a % of 2015 turnout

% voting Labour in poll

% voting Conservative in poll

% voting Green in poll

ALL

81%

66%

123%

34%

35%

6%

18-24

64%

43%

149%

41%

32%

11%

25-34

78%

54%

144%

39%

32%

8%

35-44

77%

64%

120%

38%

29%

8%

45-54

86%

72%

119%

36%

32%

6%

55-64

84%

77%

109%

32%

33%

7%

65-74

88%

78%

113%

26%

45%

4%

75+

92%

123%

27%

44%

0%

As we can see, implied turnout is higher than in 2010 for all age groups but particularly so for those under age 35 who are the most pro-Labour. This means that these groups form a larger share of our poll’s “voters” than they should. This group is also the most pro-Green although the Greens are overstated by all groups relative to their actual share so this only addresses part of the problem there.

Let’s look also at socio-economic grade. This is trickier as the classification can vary depending on who is doing the classifying but, even with this caveat in mind, it still looks like our poll is over-representing the more pro-Labour groups of the population.

Socio-economic group

Implied turnout in our poll

2015 turnout in age group (source)

Implied turnout as a % of 2015 turnout

% voting Labour

% voting Conservative

% voting Green

ALL

81%

66%

123%

34%

35%

6%

AB

85%

75%

113%

28%

40%

7%

C1

81%

69%

117%

35%

37%

6%

C2

80%

62%

129%

35%

31%

8%

DE

78%

57%

137%

41%

29%

5%

The figures for vote share and likely turnout are all determined by asking them of respondents directly before weighting such as party propensity kicks in. Whether this disparity is due to social desirability bias towards voting, or the fact that polls generally are answered by a more politically engaged section of the population, the fact remains that direct questions alone are clearly not enough to accurately isolate the voting population.

Looking again at our final poll

Taking the biases exposed above we have adjusted the numbers in our final poll to see what effect correcting for them would have had.

Going back to how we originally put it together, our voting intention polls go to a selection of respondents on our large consumer panel designed to be nationally representative according to a number of demographic factors. The sample that we ultimately achieve tends to be very close to these targets but we then use weighting to make the last few adjustments to match them.

We then apply party propensity weighting. The full explanation of this is here but in essence it makes sure that our sample is representative politically as well as demographically by asking voters how likely they are to ever vote for each party on a scale from 1-10 and from that we put them into categories such as “Labour – lean right” or “Conservative – lean left”. We know how large or small each of these groups should be and so we can weight our sample to ensure the correct balance.

In the table below you can see the original raw figures and the effect that each stage of weighting and adjustment had on them in our final poll:

Final Opinium voting intention poll

Raw figures

With demographic weighting

With demographic and party propensity weighting

Final Opinium voting intention poll

Raw figures

With demographic weighting

With demographic and party propensity weighting

Conservative

34%

33%

35%

Labour

33%

33%

34%

UKIP

13%

14%

12%

Liberal Democrats

8%

8%

8%

SNP

5%

5%

4%

Green

5%

5%

6%

The next table shows what happens if we correct for some of the biases mentioned earlier and we have made each change in stages to be as clear as possible.

We start again with the original unweighted figures and then add demographic weighting with the 7-way age split (18-24, 25-34, 34-44, 45-54, 55-64, 65-74, 75+). This corrects for the under-representation of those aged 65+. We then add turnout corrections to make our ‘voting population’ match the real one and remove the over-representation of groups like 18-24 year olds and DE voters. Finally we add our party propensity weighting.

Final Opinium voting intention poll

Raw figures

Demographic weighting with 7-way age split

Demographic weighting with 7-way age split AND turnout corrections

Demographic weighting with 7-way age split AND turnout corrections AND party propensity weighting

Final Opinium voting intention poll

Raw figures

Demographic weighting with 7-way age split

Demographic weighting with 7-way age split AND turnout corrections

Demographic weighting with 7-way age split AND turnout corrections AND party propensity weighting

Conservative

34%

33%

34%

37%

Labour

33%

33%

32%

32%

UKIP

13%

14%

15%

12%

Liberal Democrats

8%

8%

8%

8%

SNP

5%

4%

4%

4%

Green

5%

5%

5%

6%

Conclusions

It is the easiest thing in the world after an election to take your final prediction, tweak it here and there until it looks like the actual result and then claim that this is how to do it in future. That is manifestly not what we are doing here.

The adjustment that has had the most significant effect has been to include the turnout corrections, themselves based on the assumption that the 2015 electorate would have similar patterns of differential turnout to the 2010 electorate. Applying these to future elections is potentially problematic.

USpollsters construct intricate likely-voter models using similar approaches to the above and which use demographic information and past elections to predict likely turnout. But this can be problematic if the composition of that voting population changes as a great deal of weight is then put on what predictions a polling company might make about that composition. Given the perceived closeness of the 2015 British election most expected turnout to be higher than in 2010, perhaps as high as 70%. In fact turnout barely increased by one percent. This means that applying a turnout filter to match the 2010 voting population would have been more accurate but we only know this after the fact. 

Adam Drummond

Navigation

Social Media