Can Twitter "Polls" Predict Newspoll Changes? (Interim Results)

It's an almost annual tradition on this site to release something every Christmas Day. Click the Xmas tag for previous random examples.  Why do I do this?  Partly it's a present for those who like Christmas, which in a very low-key non-religious fashion includes your host, but it's also a present for those who don't want to deal with this particular Christmas or even generally cannot stand Christmas and just wish it was a normal day when normal things happened.  And what could be more normal than niche meta-psephology?  Therefore, the campaign against compulsory Christmasing brings you ... whatever this is.  

Over the past few years I've been running opt-in Twitter "polls" for the amusement of my followers there, similar to the Not-A-Polls in this site's sidebar. 

This started with the October 2020 Budget, because Budgets attract speculation about Budget bounces in party polling, but these bounces only rarely actually occur.  

The "poll" got this one wrong.  The Morrison government's 2PP rose by one point from 51 to 52.  

Over the next several months I kept posting these intermittently, then from June 2021 I came up with a stable format.  In general the "polls" would ask about the 2PP, the Prime Minister's netsat, the Opposition Leader's netsat and the PM's Better Prime Minister lead.

(Click for larger clearer version).  The format I generally use is that for the 2PP if the government is behind the options are up 2 or more, up 1, no change, down.  If the government is level or ahead the options are up, no change, down 1, down 2 or more.  For the leader ratings the options are up 5 or more, unchanged to up 4, down 1 to down 5, down 6 or more.  This is all to deal as best I can with one of the limitations of Twitter "polls" - that they only allow for four options (five would be so much better).

Sometimes I ask extra questions, eg about Budget ratings or the Voice or who will have the better netsat rating.  These are not included in this review.  

Of course Twitter "polls" are not polls at all.  They're not even limited to my followers but at the mercy of at times being retweeted by high follower left accounts into their follower bases.  I've frequently added disclaimers to this effect and promises to eventually write something up.   

And here the interim results are!  (I intend to do a further writeup in a few years time.)  Do these Twitter "polls" have any value at all in predicting what will happen in Newspoll?

Polls included

My sample includes partial results for the Newspolls released on or about 6 Oct 2020, 30 Jan 2021, 21 Feb 2021, 25 April 2021 and 16 May 2021.  From 6 June 2021 it includes full results and includes every poll up to the start of the 2022 campaign except 13 Feb 22.  For the campaign it includes just 8 May and 20 May, then after the election 30 Oct 22 and the following in 2023: 5 Feb, 5 Mar, 14 May, 25 June, 16 July, 24 Sep, 26 Nov, 17 Dec.  Whether I run it or not has depended on (i) time available (ii) whether I succeed in predicting when a Newspoll will be out!  

For each poll I've counted the results as falling in four bins 1-4 where 1 is the best available option for the party and 4 is the worst, and I've also checked which bin the actual result landed in.  As a main indicator I've then compared this with the average prediction.  So for instance if 10% predict bin 1, 20% bin 2, 50% bin 3 and 20% bin 4, then the average prediction score is .1*1+.2*2+.5*3+.2*4=2.8.  For a leadership score that range would mean 70% expected the leader's rating to worsen, and only 30% expected it to either stay the same or improve.

How did the "polls" go?

Overall my sample includes 103 predictive attempts from 24 "polls" where all four questions were canvassed and five early "polls" where some questions were canvassed but not others.  

The plurality pick was correct in 38/103 cases (36.9%).  That's a lot better than guessing, but it also isn't that impressive by itself.  After all anyone vaguely familiar with Newspoll history will know that changes of 2% or more in the 2PP are pretty rare these days (yes it's happened three times in a row as I write, but that's the first time since the old bouncy Newspoll days back in 2015).  Overall 35 of the 103 outcomes landed in bins 1 or 4 so simply tossing a coin between bins 2 and 3 (one of which always contains the no-change result) would on average have scored 34 correct picks.  On the other hand, the random noise element in polls means we shouldn't expect miracles here; if the plurality managed to get, say, 60% right, that would be pretty amazing.

There wasn't any obvious difference in the optimism or pessimism of the predictions by party, which is surprising.  The table below shows the average prediction for each group compared with the average result:

A plus score in the Difference column means that side tended to do worse than expected, a negative score means better.  On both the 2PP and Opposition Leader netsat the average predictions and outcomes were very similar.  On both Prime Minister netsat and to a lesser degree Better PM lead, the party in power tended to underperform compared to the Twitter predicition - in both the case of the Coalition under Morrison and that of Labor under Albanese.  The predictions at this stage therefore don't have any partisan house effect.

The leadership "polls" underestimated the edge results (Bin 1 and Bin 4).  Across these as a whole, the average vote for Bins 1 and 4 combined was 18%, but results landed in these bins 24/75=32% of the time.  The biggest failure was the 25 June 2023 Newspoll:

Here a slim majority correctly predicted that Albanese's net rating would go down.  But only 2.6% correctly predicted it would fall by more than five points (it fell eight points to +10).  This sort of thing may be because the audience are not that aware of how bouncy Newspoll leader ratings are.  Historically the proportions of PM netsat results in bins 1-4 respectively is 22.5%, 30.6%, 25% and 21.9%, so if a PMs net rating is going to go down, it is almost as likely to go down by more than 5% as it is to go down by 1-5%.  As a result if a punter can identify something that looks very likely to drive the PM's ratings down, then bin 4 may be the best bet.

On the other hand, the "polls" are relatively good at predicting bin 4 results on two-party preferred.  2PP bin 4 (which is sometimes a one-point drop and sometimes a drop of two points or more) happened eight times in the sample so far.  Three were tipped by a majority of voters and a fourth by a very near majority.  In all three cases where a narrow plurality wrongly predicted bin 4, the correct answer was bin 3.  In three of the four cases where bin 4 wasn't predicted by plurality, the plurality predicted bin 3.  (The only bin 4 result that was completely missed was in the early days before I standardised the format, 30 Jan 2021, in which only 6.9% correctly predicted a Coalition drop from 51% 2PP to 50 or less (it was 50).)

Overall, the average prediction landed in the right half of the list 65 times and the wrong half 38%.  That 63% rate of landing in the right half might sound pretty good, but not necessarily (eg with 2PP whichever half has the no-change option is historically more likely to be correct than the other half).  To look at whether the average prediction has any kind of predictive value it's necessarily to compare it with the correct answer.  And that looks like this:

It's not a great relationship but it is there - in this first sample, when the Twitter "poll" predicts the government or a leader will do badly, they actually are more likely to do badly.  It's statistically significant given the sample size of 103 readings but still only weakly so if I treat the sample as the number of individual Newspolls sampled (which is probably the correct thing to do).  So, more numbers needed before we can be too confident that Twitter "polls" have even the slightest idea, but at the moment it looks like they do.

In terms of the individual indicators, the "polls" so far are best at predicting 2PP change (32% of variation explained) followed by PM netsat, but have so far been useless or worse (weak slope going the wrong way) at predicting changes in Better Prime Minister lead.  This is not too surprising; Better PM is a silly compound indicator that gets far too much attention and that is naturally difficult to predict because it is influenced by so many different things.  But it will be interesting to see if this changes with more data.

That's not to say these Twitter "polls" are a very good predictor of Newspoll, or even as good as any others, just that they do appear to be better than nothing.  Historically one of the best predictors of 2PP is this: that when the Newspoll 2PP moves, it will probably not move the same way again next time.  When a government's 2PP polling has gone up, 52.5% of the time it has gone down in the next poll compared to 26.3% up again and 21.2% no change.  When a government has gone down, 52.3% of the time it has gone up next time, compared to 22.7% down again and 24.9% no change.  (No change has been followed by a rise 36% of the time, a fall 40.4% of the time and no change 23.6% of the time.)

Since the outsourcing of Newspoll from 2015, it has been less bouncy than before, but this pattern is still there: a change has been followed by a change in the opposite direction 42.9% of the time, in the same direction 18.7% and no change 38.5%.  However the commonest sequel to a no change result has been another no change result (40.7%) followed by a fall (33.9%) and a gain (25.4%).  However there have been a few significant changes in Newspoll over even this time: the 2015-9 mixed online/robocall Newspoll was notoriously stable, the 2019-23 YouGov online Newspoll a little less so, and the new Pyxis Newspoll so far may be a little less so again.  

One reason for this, which runs contrary to common gallery nonsense about polling "momentum", is that most of the time what is actually going on in voting intention is nothing, so most cases of 2PP movement are random movements that are more likely to be cancelled out the next time.  Another is that some factors may influence voting intention but only last (at full strength or at all) for one cycle.

Can Twitter "Polls" Predict Findings About Twitter "Polls" About Newspoll?

Earlier today I put the above findings to a series of Twitter "polls" themselves!  The form in which I did so was a but suboptimal since I forgot to mention at the start that I was referring to Twitter "polls" on my account only (that said nobody else seems to have run all that many and I think most respondents would have been familiar with my Newspoll-night cycle).  Also we have no way of knowing how many respondents were sober.  How did it go?  Not too well!

The plurality selected "useless" but in this sample at least the correct answer was "somewhat".

I would have guessed Labor in advance of doing the article but in this sample the correct answer was "neither".  This may reflect my follower base; polls by a random Twitter occupant probably would skew to Labor.

The majority got this one correct: 2PP was the best predicted indicator in this sample.

The 2PP option polarised respondents: those who didn't think it would be the best predicted tended to think it would be the worst predicted.  However almost no-one (7 voters) correctly picked Better PM lead as the worst predicted, though I thought that was the obvious choice because of its compound nature.

For this one at one stage in voting, the correct answer (less often) was catching up, but then "more often" pulled away.

Secular seasons' greetings to all and best wishes to all readers for 2024.