How Did The Super Saturday Seat Polls Go?

With final results for the Super Saturday by-elections now available, including preference flows, it is worthwhile reviewing the accuracy of the seat polling for these by-elections.  Recently I have posted a few times here about seat polling - see Is Seat Polling Utterly Useless? and Why Is Seat Polling So Inaccurate?  The preferencing results of the Longman by-election are also of special interest to the debate about Newspoll preferences, as the first test of Newspoll's changed methods at any federal election.

We can put that one to bed right away: the preference flow from One Nation to the Coalition in Longman was a massive 67.74%.  This should not be taken as a sign of quite how strongly One Nation preferences would flow nationally, since they did preference the LNP in this seat but would not necessarily do so in every seat, and since LNP preferences flowed strongly to One Nation in the area at the Queensland election even in the seat where the party preferenced Labor.  But it does show that it is entirely reasonable for pollsters to assume that 2016-election preferences can't be trusted in the case of One Nation.



While Newspoll's approach was vindicated on that score, the seat polls for Longman were otherwise pretty bad.  It was the most polled seat so I'll start there.

A few notes about my accuracy assessments - I am just using average error rather than the technically fancier Root Mean Square Error.  Partly this is because I'm seeing average error used more often in other Australian comparisons and partly it's because in a case where a pollster, say, overestimates both majors and underestimates Others, RMS can be especially harsh on a mistake that might not make any difference to the outcome.

In the case of ReachTELs I've redistributed their so-called "undecided" value (soft voters who any other pollster would put in the headline), using the breakdown by party where I could find it, proportionally where I could not.  For each seat, the 2PP (act) column shows what the 2PP would have been based on the preference flows recorded had the pollster's released primaries been exactly correct - this is useful for seeing if the pollsters got the preference flows right. The Error (P) column is the average of primary vote errors, and the Error (2) column is the average of Error (P) and the error in the pollster's released 2PP estimate.

Some of the categories were omitted from some of the polls and in those cases the Others figure includes those categories and appears in italics.  Finally I do not include the 2PP data for Fremantle because both pollsters expected it to be a Labor-vs-Green 2PP but it ended up being Labor-vs-Liberal Democrat.

Longman

Polling generally failed in the Longman by-election, with Newspoll scraping a pass at the end by at least getting the right winner (unlike all the other polls).  Otherwise, this was pretty much a repeat of the mass poll strandings that occurred in Macarthur, Bass, Dobell and Lindsay at the 2016 federal election.  A suspiciously hedged (if not herded) looking set of 2PPs missed the mark by an average of 5.1 points. The primary source of the two-party error was that all the polls overestimated the LNP primary vote, by an average of 7.0 points (which would be outside the in-theory margin of error if considered individually, let alone as an average across a set of polls).  The beneficiary was Others, who polled 9.81 points between them, more than double the average estimate in those polls that carved out One Nation! But that wasn't all - even had the polls got the primaries right, they were overestimating the flow of Others preferences to the LNP, and hence had the Labor 2PP on average 0.9 points too low for the published primaries.

That's still much closer to the correct preference flow than had they tried to use generic last-election preference flows to model the 2PP, but tragically the pollsters actually made their 2PP forecasts worse by improving their forecast of preference flow.  Had they gone with last-election preferences in this case, the error in so doing would have helped mitigate the error with the LNP primary vote, and every Longman poll would have predicted a winning Labor 2PP.  Had that been the case, right-wing expectations of success in Longman would not have been so high, and perhaps Malcolm Turnbull would still be Prime Minister (though given the major parties' fixation on the primary vote in polling, one might doubt it.) 

Braddon

In Braddon we also saw pollsters get the Others vote much too low, but in this case they had both major parties too high, so the error had much less effect on the 2PP.  In this case there was no significant issue with preferencing and the last few polls were pretty good.  The early polls had the Liberal primary much too high, but it is possible that this represented a genuine afterglow of the party's state election win.  The Australian Forest Products Association ReachTEL was easily the most accurate poll of this by-election (even if one adjusts it on account of the supplied 2PP being apparently incorrect on the basis of the published data - it looked like it should have been 51 to Labor.)

Independent fisherman Craig Garland's preferences flowed 74.31% to Labor (slightly stronger than the preferences of the Greens, on account of the latter including the donkey vote).  There's evidence from booth data that about a third of Garland's voters switched their preferences from Liberal to Labor at this by-election. While those voters might have switched anyway, I think there's a strong case that Garland's run and the campaign fight between him and the Liberals at least killed the Liberals' chances of making Braddon very close, and may even have decided the seat.

Mayo

If Longman was a seat-polling failure, Mayo was a seat-polling success.  Before there was any polling on Mayo, the narrative was that Rebekha Sharkie would be affected by the state election failure of her party (variously Nick Xenophon Team, SA-Best and Centre Alliance) and that if her vote declined enough she could easily lose.  She was at $3 to retain in betting.  Polling showed that Sharkie was in fact a hugely popular local member irrespective of the fortunes of her party and was cruising to victory.  Even had Sharkie only won, say, 53-47, the polls would have still been more predictively useful than had there been no polls at all, but in fact her magnitude of victory was almost as large as the polls expected (with one ReachTEL overcooking things significantly because it used respondent preferencing, which is volatile.)

Fremantle

There were just two polls of Fremantle.  The Legalise Vaping ReachTEL overestimated the Liberal Democrats and underestimated Others, but was still reasonably good.  The YouGov Galaxy reported primaries only in the form Labor, Green and Other, and perhaps if that was the form it asked questions in that would explain why it had the Green vote much too high.

Perth

There was only one poll of Perth, another Legalise Vaping ReachTEL, and unlike the Fremantle ReachTEL it was way off most of its marks.  I have some sympathy for the pollster here - it's not easy to predict how voters will respond to a 16-candidate ballot paper, with 36% of voters not even turning up and 10% of those who do turn up voting informal!

Overall

The average error in primary vote estimates for these by-elections was about 3.1 points.  That's still not great, but it's a lot better than the average error seen in seat poll primaries (around 4 points) for the 2016 election.  The average 2PP error was around 3.5 points, which was slightly but not significantly worse than at that election.  The effective margin of error is about twice these averages, so the by-election has hardly seen the problems with seat polls fixed.  Rather, as was also the case in the federal election, we saw generally good results in some seats but a general failure in the most heavily polled contest.  Once again, we would hope that in the most heavily polled contest at least the average would be somewhere in the ballpark, but something went wrong with the primary vote polls in Longman, and hence it wasn't to be.  Perhaps there was some late swing away from the LNP as a result of robopoll bombardments, but I for one do not believe the LNP primary sat in the mid to high 30s for months before falling off a cliff in the final days.