2019 Federal Election: Pollster Performance Review

Welcome (belatedly) to another of my regular pieces that I do after all the election results are finalised and, um, we can't really give this one its usual title this year.  Normally it's called "Best and Worst Pollsters" (see the comparable articles for 2013 and 2016) but this year that title isn't really appropriate.  This year was the year of the great Poll Fail, and when it came to final voting intention polls at least, they all went down together.  The story for seat polling turns out to be a little less clearcut, but not that much.

For all the complaints about "too many polls", the frequency and diversity of Australian polls had been declining at state and federal level in the four years leading up to this election.  At this election there were only five poll series conducting national polls, and of these two were conducted by the same pollster (YouGov-Galaxy conducts both Galaxy and Newspoll polls).

I usually include three categories but this time I'm not going to take tracking too seriously.  As usual the first cab off the rank is ...

Least Worst Final Poll

I usually say the final poll should be the easiest one for the less accurate pollsters to get right, because pollsters can look over each others' shoulders and consider corrections if everybody else is getting something vastly different.  Thus there have been some prior cases where polls that differed from Newspoll for some time have jumped into line with it in their final poll.  This year unfortunately it seems that some pollsters may have taken this concept a little too far - either that or multiple pollsters got to around the same 2PP coincidentally and then decided to self-herd from that point.



Again, I've used root mean square error (the RM(2) column) to assess the accuracy of the polls - the lower the better.  There is also a column just for the error on the primaries (RM(P)).  Again I've included the 2PP figure, with a weighting of five so that errors in the 2PP make as much difference as errors in all the primaries put together.   Here's the ranking (click for slightly larger/clearer version):

L = landline, M = mobile.  The Oth (U) column is Others including UAP - Essential did not break UAP out of Others.  The Prefs column is the share of minor party preferences that would have been expected to go to the Coalition based on the published 2PP.  Closest readings to the pin are in bold and anything within 1 point of accurate is in blue.

As is their habit, Ipsos had the Labor vote much lower than other pollsters and the Green vote much higher.  This mainly resulted in them being right on the Labor vote and wrong on the Greens vote, whereas most of the other pollsters were the other way around.  But Ipsos were also not as wrong on the Green vote as the others were on the Labor vote.  Ipsos were also one of the closest on the Coalition vote, and had the One Nation, UAP and Others votes fairly close.  As a result Ipsos has come through as having the least worst final poll.  However, it has done so with a score worse than that of all bar one pollster in 2013 and two in 2016.  For its efforts Ipsos wins an unceremonial public dumping by Ninefax.

Galaxy and Newspoll didn't make significant errors on preferencing overall (indeed they actually had the Coalition getting slightly more minor party preferences than it did, as a result of said polls achieving the Herculean feat of having the Green vote too low).  Their main issue was simply having the major party primaries, especially Labor, wrong.  Morgan actually did better on the primaries than both the Galaxy products and as well on them as Ipsos, but worse on the preferences, and Essential had One Nation much too high.  I suspect the latter problem resulted from failure to screen out seats Essential was not running in.

Note added 2021 because someone is being very silly on Twitter: 

Across an average of all polls underestimating of preference flows contributed 0.2%-0.6% to the overall 2.9% error on the 2PP, depending on how the contribution is measured.  These errors varied by pollster as follows:

If measuring by overall preference flow:

Ipsos: preference assumptions 1.1%, wrong primaries 1.4%

Galaxy: preference assumptions -0.4% (counteracted error), wrong primaries 2.9%

Newspoll: preference assumptions -0.5%, wrong primaries 3.5%

Morgan: preference assumptions 0.9%, wrong primaries 2.6%

Essential: preference assumptions 0.1%, wrong primaries 2.9%

If measuring compared to actual preference flow by individual minor parties (probably best):

Ipsos: preference assumptions 0.7%, wrong primaries 1.8%

Galaxy: preference assumptions -0.2%, wrong primaries 2.7%

Newspoll: preference assumptions 0.1%, wrong primaries 2.9%

Morgan: preference assumptions 1.3%, wrong primaries 2.2%

Essential: preference assumptions 1.0%, wrong primaries 2.0%.

Tracking

In the past two elections I have recognised the pollster that best seemed to track the development of voter intention towards the final result.  This election it's not really possible to do this, because not only were the voting intention polls in general wrong, but we don't know at what point they went off the rails.

As a very rubbery approximation I can assume that the polls were in general wrong by the same amount through the campaign and then measure the average voting intention error against a picture in which the Coalition starts in the mid-50 range, rising to the mid-51 range at the end.  On this basis using released 2PPs as a yardstick Newspoll (average error 2.56) comes out trivially ahead of Morgan (2.59), Galaxy (2.63), Essential (2.81) and Ipsos (3.16), with Ipsos being marked down because of strong results for Labor in the earlier stages.  The Galaxy stable polls only come out as slightly better on this indicator because of their better handling of preferences compared to Essential and Ipsos.

Seat Polls

The accuracy of seat polling has been a hot topic at this election.  Seat polls were very inaccurate in 2013 (when they skewed heavily to the Coalition, which the national polls didn't) and weren't that good in 2016 (when they skewed more modestly to the Coalition but were greatly under-dispersed compared to the actual distribution of swings.) YouGov-Galaxy's David Briggs has tried to limit the 2019 damage by pointing out that Galaxy seat polls correctly showed Labor failing to pick up seats.

I dealt with this to some degree here.   Galaxy seat polls in the final week were uncannily precise in picking the correct winner when they picked a winner at all, but they were lucky, because they were every bit as skewed on average as the national polls.  There were also several seats that Galaxy had at 50-50 that the Coalition won massively, and the seat polls in seats that were polled more than once were suspiciously under-dispersed (also the case in 2016).

Unfortunately YouGov-Galaxy was the only pollster to release significant numbers of seat polls at this election, and that applies even after scraping the barrel by including the very few polls commissioned by lobby groups that saw the light of day.  Only six non-Galaxy seat polls were released and only one of those was media-commissioned.  (There were also claims about internal polling in specific seats, but there is no evidence that those were all even genuine.)

In assessing seat poll accuracy at a two-party or two-candidate level, I've adopted a few conventions over the years.  One is that a 50-50 only counts as a hit if the result is genuinely close (52-48 or closer either way), and another is that the "easiness" of the seats being polled is estimated based on the results, so that pollsters who mainly pick easy seats don't look good just by getting those right.  The following is this year's table (I've included five lobby-group or detailed internal polls this time just because the cupboard would have been very bare without them.):

Polls are in alphabetical order.  This is not a ranking.

I've bracketed two uComms-using-ReachTEL polls (Bass, Braddon) with one ReachTEL poll that may or may not have been a uComms as well (Corangamite); unclear media reporting on this is common.  The skew column shows the average 2CP skew to Coalition on the two-party (or in cases two-candidate) figure reported; a negative figure is a skew to non-Coalition forces.  The Error column shows the average 2CP error irrespective of direction.  The Correct column shows the percentage of polls that were correct.  The Ease column measures how easy or difficult the seats polled were - a low figure indicates close seats, while if the seats are all very lopsided the figure will be 100.

Galaxy-branded polls were marked as correct for all 16 cases in which they picked a winner, but wrong for all five 50-50s as none of them were close.  For instance a truly random sample of the seat of Forde taken in the final days would have picked the Coalition as winners over 99% of the time.  The table shows that the seat polls conducted by Galaxy (whether under that name or as Newspoll) were severely skewed to Labor, as was their public polling.  The few uComms/ReachTEL seat polls did not show such skew (in fact skewing to the Coalition on average).

Lonergan's GetUp! commissioned poll of Warringah was rather accurate.  Environmental Research and Counsel's seat polls for the Greens continued the long record of released Greens internal polls being too favourable for the Greens.

It's hard to make any award based on a sample of just three polls but the performance of the uComms/ReachTEL polls does leave us with an intriguing what-if regarding the absence of the ReachTEL platform, or indeed any pure robo-poll, from the national voting intention scene.

Senate Polling

I should also mention the Senate polling released (in inadequate detail) by The Australia Institute and conducted variously by Dynata (Research Now) and Lonergan.  Unfortunately the May final model was not posted anywhere on the net that I can find, despite being reported by one of TAI's usual clearing-houses.  The March model projected the Coalition on 28% primary (actual 38), Labor 33 (28.8), Greens 12 (10.2), One Nation 8 (5.4) and UAP 2 (2.4).  It's not clear how much the primaries moved before the May polling was taken, but I do know that the overestimation of One Nation continued right until the end, with the party said to be on for a ludicrous 9.5% in Tasmania (it got just 3.45%).  Senate polling has had an even worse reputation than seat polling and these results should do nothing to repair it.

Correction note: The main table was reposted on 7 Nov 2019 to correct a minor error in recording of the Morgan primaries, with resultant recalculation.  This did not alter the ranking order.