What Might 2PP Voting Intention Have Really Looked Like In The Last Federal Term?

(An update was added to this article on 2 July 2020 - scroll down.  The original text has not been revised.)

The 2016-2019 parliament saw Australia's worst failure of national opinion polling since the early 1980s, a failure that was not just a combination of normal errors and a reasonably close election.  Aggregated polling had the Coalition behind for the entire term, at no stage better than 49% two-party preferred, and yet the Coalition won with 51.53% of the two-party preferred vote.

The view that the polls were in fact right all along but voters changed their minds at the last moment (either on election day, or on whatever day each elector voted) fails every test of evidence that it can be put to.  The difference between voting intention for voters voting before election day and on election day is similar to past elections, and if anything slightly stronger for the Coalition.  There was no evidence in polling of change in voting intention through the final weeks, as would have been expected as voters who had already voted reported back their behaviour if the polls were at all times accurately capturing the intentions of the person being polled.  Also if those who had already voted had shifted towards the Coalition as they made their final decisions while those who had yet to vote were yet to do so, there would have been polling gaps of several points between those who had already voted and those yet to do so; this was not the case in the released evidence either.  

It is therefore overwhelmingly likely that the polls went off the rails sometime before the last few weeks.  But when, and did this happen suddenly, over time, or some combination of the two?  What might a polling aggregate of the term we have just seen had looked like had the polls been accurate at all times?

To start with, this may well be something we will never really know with that much certainty.  But it's worthwhile trying different models and seeing which ones are open to obvious objections and which ones are possibly OK.  

Existing Retro-Aggregations

The first retro-aggregation I have seen is in a set of conference slides by Simon Jackman and Luke Mansillo.  You'll have to use the right arrow key to scroll along to slide 9 which is the 2PP retro-aggregate.  Their assumptions include an unknown discontinuity for the Turnbull-Morrison leadership switch and also that each pollster has a time-invariant house effect.  In slide 11 they find all the pollsters to have quite large house effects on this basis (from just over 2 to just over 3 points 2PP.)

The result of the Jackman and Mansillo retro-aggregation shows the Coalition getting a small bounce up to 51-49 just after the 2016 election, then dropping back to below 50% at the end of 2016, and then remaining narrowly behind for a year and a half.  In the lead-up to Super Saturday, the Coalition briefly grabs a slender 2PP lead, then drops in the leadup to and further as a result of the spill, and falls to a term low in the high 48s as Scott Morrison arrives in office.  The Coalition climbs rapidly from late 2018, takes the 2PP lead in early 2019 and is then never headed.

Overall a remarkable aspect of this retro-aggregate is the lack of variation in the Coalition's standing.  The government sits in a roughly two-point band for nearly all of the term.  

(One minor data point I'd make here concerns the treatment of "YouGov".  They record 15 YouGov polls but I am aware of 16, these being 13 in 2017 by YouGov-Fifty Acres and three in 2019 by YouGov-Galaxy.  I would have treated these separately, as they are completely different pollsters that were operating in conjunction with (and in the latter case after being acquired by) one global polling firm.  It's also obvious from the assessment of house effects to the YouGov series that Jackman and Mansillo have ignored YouGov-Fifty Acres' idiosyncratic respondent 2PPs and recalculated 2PPs for this series.  I agree with this approach.  While it's true that Fifty Acres often showed Coalition leads, it did so off primaries that would have seen the Coalition belted.)

The second retro-aggregation I've seen was a set of 2PP aggregations posted today by Mark the Ballot.  (Also see another earlier Mark the Ballot post here.) These Bayesian models are "anchored" to one of the 2016 or the 2019 results, or both.  The 2016-anchored model (left-anchored) has the Coalition behind for virtually the whole term and sneaking over the line at the end, though the final estimate for the Coalition is lower than what actually occurred.  The 2019-anchored model (right-anchored) has the Coalition ahead for most of the term, except in the aftermath of the spill, but has them jumping to 52% in the aftermath of their 2016 win, which doesn't make sense to me (see below).  The model anchored to both sides avoids both of these issues and is generally very similar to the Jackman and Mansillo model, except that it finds a larger drop on the switch from Turnbull to Morrison.  

Retro-Aggregation With Morrison As The Culprit

I am going to present some retro-aggregation graphs that make a different assumption to any of these.  I don't claim this assumption to be an established fact, just a very plausible hypothesis.  The question I'm interested in is: what if the change of Prime Minister was a major cause of the polling failure?  

I think there are some sound arguments for this having been the case.  The previous election was contested between Malcolm Turnbull and Bill Shorten.  Final 2PP polls were very accurate at that election, albeit herded in the final week specifically.  Turnbull and Shorten continued as leaders after the election.  The polls immediately after the election made sense, because Turnbull's result was viewed by Coalition supporters as a clear disappointment, and there was no reason for him to get a polling bounce for a disappointing result.  So why should we think a house effect was suddenly present in the early polls of the term having not been evident in the late polls of the previous one?  The assumption of an invariant house effect through the whole 2016-9 term just doesn't make sense. Whether one gradually developed through the term is another question.  

Secondly, if pollsters either over-captured educated voters or under-captured disengaged voters (two likely possible causes of the polling failure) there's every reason to think they would have done so under Morrison specifically.  Thirdly, we did have some electoral events during the Turnbull term, and the Super Saturday results taken together at least didn't really suggest Turnbull's government was cruising (though they are messy to interpret because of their unusual circumstances).  

The change to Morrison was unusual compared to previous Prime Ministers.  Previous mid-term changes of Prime Minister have always brought dividends for the government changing its leader, though the benefits of the controversial change from Kevin Rudd to Julia Gillard in 2010 were small and fleeting.  The leadership change in 2018 was extremely messy and public anger concerning it was not surprising - but on the other hand Malcolm Turnbull was not a very popular PM.  

There's nothing implausible about the idea that a leadership change should produce house effects.  Indeed changes in the house effects of polls relative to each other are often seen following leadership changes (Morgan vs everyone following the installation of Turnbull, and Essential vs Newspoll following his removal).

My models below (using the same relatively simple aggregation methods as I used for the term, except for changing the global house effects) are anchored to the 2019 result at the end, but assume that the polls were accurate until Turnbull was removed.  The first one assumes that they then went haywire immediately:

In this case the leadership change (at about week 105) actually has no immediate impact on voting intention (the downswing before it is caused by the bad final Ipsos for Turnbull).  The government is competitive but trailing within weeks of Morrison's arrival and starts 2019 on level 2PP terms, taking a clearly winning lead in the campaign.

This one assumes that a house effect developed gradually once Morrison became PM:

Here we have the same polling blowout caused by the spill as in the original aggregate, but it is followed by a rapid recovery that runs (with the odd stall) all the way to polling day.

This one is a hybrid of the two, with half the polling error happening as soon as Morrison becomes PM, and the other half developing gradually:

And this is one where half happens as soon as Morrison becomes PM, and the other half happens during the campaign (perhaps being missed by the late polls because of herding):

These versions all show the government clearly in the lead only in the last month or so of its term.  They vary as to whether it was more or less equal before that - with the caveat that the strange distribution of seat swings seen at the election means the government could actually have won in minority with 48.9% 2PP or perhaps even 48.5%. On the most optimistic of these assessments, the government was actually on track to a scraped re-election from the moment Morrison took the helm; on the least it was losing until early April.  

I post these because it's tempting to conclude from the polling failure we've seen that Malcolm Turnbull was removed in an election-winning position.  But I don't think that should be accepted as hard fact.  We pretty much know the polls were wrong at the end.  We have no reason to think they were wrong at the start.  They may have been getting worse through the term, but is there a way to say where?

I hope this article will encourage other retro-aggregation attempts and discussion about sources of evidence about the point(s) at which the polling failure may have developed.

Graphs Added (2 July 2020)

The release today of "Morrison's Miracle"  has spurred me to update this article with an additional hypothetical graph.  In the Jackman and Mansillo chapter, they include a version of the same graph I mentioned earlier (the one with time-invariant house effects in which the Coalition is never losing by much.)  My view is that house effects were almost certainly not stable over the 2016-2019 term.  Why should they be, when no house effect to speak of had been present at the 2016 election?

However, something I hadn't tried was assuming a constantly drifting house effect over the 2016-9 term.  On this assumption, polls start out accurate but gradually get worse over the course of the term, for whatever reason.  I had thought this was the same as Mark the Ballot's left-and-right anchored model, but the result is different in some places.  The difference most likely relates to different treatment of the YouGov-Fifty Acres series; MTB's aggregate takes the 2PPs as published while mine re-derives them from the primaries.

Anyway, here it is:

In this model, Labor led by around 52-48 from late 2016 to early 2018.  The gap narrowed with the Coalition narrowly ahead around Super Saturday and in a probably winning position when Turnbull was rolled (with unrealistic expectations of the by-elections contributing to that), although a small 2PP lead had just been lost in the wake of that result.  The very well received Budget saw the government equalise and the campaign then saw a blowout.

I am not sure that this model is actually what happened either, but there is one piece of possible supporting evidence that house effects could have been gathering steam over the whole term.  This comes from Jackman and Mansillo's finding that the poll 2PPs were significantly under-dispersed (less variable than they should be) not just in the campaign but for the entire term.

So, here's another version, which has half of the poll failure being caused by the leadership change and half being a product of drift over time:

This one shows a slightly larger lead for Labor through to early 2018, and only a recovery to near-equality just at the point where Turnbull is removed.  It also has the Coalition suffering only a three-point hit for the leadership change, most of which is recovered immediately.