The Performance of The Polls John Curtice whatscotlandthinks

The Performance of The Polls John Curtice whatscotlandthinks. org @whatscotsthink

Why Polls Matter • Can affect the extent and nature of media coverage • Can affect the ability of campaigns to raise money and motivate activists • Can influence the campaign’s strategy and tactics • Can influence whether people vote

The Challenge • Regular monthly political polling in Scotland ended in 2003. Partly thanks to perceived poor performance, and partly the result of worsening newspaper finances. • Thereafter mostly episodic and concentrated at election times. • So companies having to estimate the attitudes of a population most have not been regularly monitoring. • And for a ‘vote intention’ that they have not previously attempted to estimate. • While the industry is still coming to terms with the internet

The Performance - 1

The Performance - 2

A Time Trend? 70 60 62 61 61 59 57 39 41 43 50 40 38 39 Yes 30 No 20 10 0 Feb-May July-Sept Oct-Dec Jan-mid Feb-Mar Based on 9 polls conducted Feb-May, 10 polls July-Sept, 8 polls Oct-Dec, and 7 polls in Jan-mid Feb and 10 polls mid-Feb. Mar. Don’t Knows excluded.

House Differences 50 45 45 43 39 40 36 % Yes 35 30 25 20 15 10 5 0 Panelbase ICM/Survation Based on all polls since Sept. Don’t Knows excluded TNS/You. Gov Ipsos MORI

Panelbase’s Record % Yes 60 51 50 40 44 45 45 44 45 43 44 47 30 20 10 0 Mar. 13 May. 13 Jul. 13 Aug. 13 Sept. 13 Nov. 13 Jan. 14 Feb. 14 Mar. 14

Ipsos MORI’s Record % Yes 40 35 38 34 34 May. 13 Sept. 13 37 36 Dec. 13 Feb. 13 30 25 20 15 10 5 0 Feb. 13

The Don’t Knows 35 30 33 30 27 25 20 15 15 10 5 0 TNS Others Source: 1 st 2 cols: Average of all polls since September You. Gov - AV ICM - AV

Methodological Differences Company Method Political Weighting/Selectio n Question Panelbase Internet 2011 Holyrood Vote; 8/10 likely to vote Was current vote; now intend to vote Sept ICM Internet 2011 Holyrood Vote Think will vote Sept Survation Internet 2011 (1 st poll: 2010) Vote now & likelihood of voting TNS BMRB Face to face quota 2011, incl abstention rate Intend to vote Sept You. Gov Internet (18+ only) 2011, + extra Lab 10, SNP 11 group Vote now Ipsos MORI Telephone (RDD) None. Figures based Vote now on certain to vote

Record Panelbase – no previous polling close to election day ICM – previously all election polling done by phone Survation – new company since 2010 TNS BMRB – still using System Three’s (not always successful) approach • You. Gov – over 10 years experience. Panellists’ previous vote collected on joining/after election • Ipsos MORI – using same approach as for GB polls since 2008 • •

How Their Last Poll Performed In 2011 % SNP Const Error % SNP List Error TNS BMRB 45 0 38 -6 You. Gov 42 -3 35 -9 Ipsos MORI* 45 0 42 -2 * Last Poll Conducted 14 -17. 4. 11

The Extent and Impact of Weighting Company Men 16 -34 C 2 DE Impact on % Yes Panelbase 1. 16 1. 17 1. 40 +2 ICM 1. 03 1. 34 1. 33 +3 Survation 1. 17 1. 42 1. 01 +3 TNS BMRB 1. 03 1. 15 0. 96 +3 You. Gov 0. 90 1. 21* 1. 26 +1 Ipsos MORI 0. 98 1. 08 - +2 * Based on 16 -39

Reported Turnout 90 % certain to vote 80 70 74 73 65 60 79 73 64 54 57 78 69 63 68 56 52 54 50 40 30 20 10 0 Feb May Referendum Source: Ipsos MORI (Scottish & British Polls) Sept Holyrood Dec Westminster Feb

Polls of The Interested? % DNV etc 2011 40 37 35 30 25 24 27 29 20 20 15 10 5 0 Panelbase ICM Survation & You. Gov figures may include a few Other party voters TNS BMRB You. Gov

Tentative Conclusions • Particularly large house effects on % Yes vote • Do not simply correspond to internet vs noninternet • But do correspond to experienced vs. less so • Internet samples more heavily weighted • But weighting etc has pushed all recent polls in a pro-Yes direction • Some polls (at least) look set to be wrong – but which ones?