Following that announcement, a further three Newspolls were released using the same method mix that succeeded spectacularly in 2016 but failed badly in 2019. This appeared to be an exercise in going through the motions while the new poll was set up and there was no reason to take those polls all that seriously. Now the new version is out, together with some new comments about the poll's direction.
The comments can be mainly found (paywalled) in an article from YouGov's Head of Public Affairs and Polling Campbell White, and also in the news report of the poll. (There was also some commentary in the editorial, but it didn't add anything.) New comment or information in these reports includes:
* "Unfortunately, response rates for robocalls in Australia have been falling and many people find them annoying and invasive. They tend to be answered largely by older people or those who are very interested in politics. Busy people who are less interested in politics either don’t answer or hang up. We believe this was a significant contributor to the inaccuracies seen at the election." (White)
* "The more complex scripting we can do online will also allow us to tailor our questionnaires more precisely to reflect the choices voters actually face at the ballot box." (White)
* Data will be weighted by "age interlocked with education" (for example; household income is also mentioned) and quotas will be applied by electorate type.
The ALP Review
There was some further, albeit murky, insight into the polling failure in the Australian Labor Party's review of its 2019 defeat. The party received a submission from YouGov regarding the internal tracking polls that YouGov conducted for the party. The review notes that:
"The submission from YouGov identifies the error was primarily attributable to a reliance on 2016 vote recall as a weighting factor. That is, YouGov’s surveys included a question asking participants who they voted for at the previous federal election in 2016, and the final report produced from each survey was weighted to adjust for any over or underrepresentation of past support based on the actual results in 2016. While that has been a tool used by quantitative researchers for decades, it served to both overestimate the Labor primary vote and the preference flow to Labor."
This was rather startling news since "past vote" weighting has been in the doghouse in Australian public polling discussion for some time. Lonergan were criticised, including within the industry, for using it in some very inaccurate seat polls taken for the 2013 election so I wasn't expecting to see that major pollsters were using it in public or internal polling.
To summarise the problem with "past vote" in an artificially simple example, suppose that a pre-election sample of 1000 voters would have had a 2PP vote of 51.5 to Coalition after all other weighting was applied, but that 45% of the sample (after other weighting) would have said they had voted 1 for the Coalition in 2016 while 32% would have said that they had voted 1 for Labor. The pollster applying past vote weighting would in effect conclude that 2016-election Coalition voters were now over-represented in their sample and that it needed reweighting to fix this. After this reweighting, the sample would show Labor winning.
But suppose that what was actually happening was that 3% of the sample said they had voted Coalition in 2016, but had actually voted Labor, and didn't want to admit to it for some reason. In that case, weighting by past vote would effectively change an accurate result after the other weightings to an inaccurate one.
Unfortunately the review is rather vague on what public polls were weighting by past vote. It only says "This error was perpetuated amongst a range of published polling at the same time." It seems likely that the Galaxy/Newspoll public polls were also involved, but this is unconfirmed.
One issue I have with all this is that unless you have been longitudinally tracking voters and finding that their recall of past vote has drifted (as YouGov have done in the UK), how would you know that past vote was the cause of an error? (And of course, if you knew for sure there was drift, you wouldn't weight by current recall.) That is, past vote might have such an effect that if you didn't weight for it there would be no error, but it might still be the case that the voters were recalling their past votes accurately and the real error was somewhere else. If the pollster oversampled politically engaged voters in some way weighting couldn't detect, then it could be that previous-election Coalition voters in the sample were more likely to defect to Labor, but were not representative of Coalition voters as a whole. However the use of risky weightings such as past vote can be added to failure to capture disengaged voters and overcapturing of politically engaged and highly educated voters as major contenders for causes of the polling failure (for why I discount some others, like late swing and Shy Tory effect, see an earlier piece.)
Past-vote weighting seems to me like something pollsters would do if they were not confident they had the weightings or quotas they were using right and feared that there was a massive political skew that they weren't capturing. So long as the number of voters who misrepresent their past vote is modest, past-vote weighting should protect polls from enormous errors. However it increases the chance of a fairly big error.
[I've been considering writing a full article on the ALP review but have not yet done so. I should mention a few quibbles in passing though:
* The review's explanation of Bill Shorten's unpopularity as a factor ultimately relies on "focus-group research conducted by Essential Research involving groups of swinging voters, who were concerned with climate change and were contemplating voting Labor but decided to stay with the Liberal Party, identified leadership as a key reason for not switching." Well of course they would say that rather than admit they chickened out of voting Labor for reasons of self-interest despite their professed climate concern, wouldn't they? And in any case focus groups are just bad polls with very small sample sizes that don't correctly simulate how people make voting decisions.
* The review blames One Nation preferences for Labor's defeat in Braddon, but Labor lost Braddon so heavily that had they got every single One Nation preference in that seat (which One Nation could not have got near delivering to them if it tried) they would still have narrowly lost. In my rear-vision window view, Labor really lost Braddon largely because of (i) Scott Morrison's superior appeal to Malcolm Turnbull in such areas (ii) valid perceptions that Labor's tourism strategy was south-focused (iii) having a better candidate than the previous incumbent.]
The New Poll's Properties
Unfortunately at this stage the new Newspoll has not arrived with any form of detailed published crosstabs, or with something that is a clear statement of all the weighting/quota selection methods used. The official line seems to be that YouGov wants to establish an Australian Polling Council (something Essential are also on board for at least in theory) and have it determine what standards are appropriate. So we still don't have anything like the level of transparency that we could, though we may see improvements down the track.
There is nothing in the voting attentions that would immediately draw anyone's attention to a method shift. The 2PP has only moved by a point (back to 51-49 Coalition after spending one poll at 50-50) and all the party primaries are within their recent range. Of course it might be that the methods change has masked a shift in the other direction, but there is no evidence of this.
What the poll does show that is unusual involves the ratings for the individual leaders. Both leaders have lost twelve net satisfaction points since the previous poll, with Scott Morrison (43-52) on a personal worst of -9, and Anthony Albanese (38-45) back to -7 after going up 12 in the poll before for no readily obvious reason.
It's not that unusual for a drop of twelve points or more to occur for a leader in a single poll; this has happened in 6.5% of all Newspolls for Prime Ministers and 3.7% for Opposition Leaders. However, there has only been one prior case of both leaders losing 12 or more points in the same poll. That was in February 2003, when for whatever reasons John Howard dropped 14 points from +19 to +5 and Simon Crean dropped 12 from -10 to -22. Howard was in a minor lull between a khaki bounce from the 2002 Bali bombing and another one from the 2003 Iraq invasion, while Crean never really recovered.
One might suggest there were reasons why Morrison's rating might have slumped, such as perceptions of his government's response to climate change in the wake of the NSW bushfire disaster that has so far claimed six lives and several hundred houses. However, I suspect that those who tend to blame Morrison for that situation mostly don't like him anyway. The recent Scanlon Foundation report found that the proportion of voters nominating environmental issues (including water shortages) as the most important issues had jumped from 10% last year to 19%, but that this was heavily polarised by party with 54% of Greens and 21% of Labor supporters holding this position compared to only 7% of Coalition voters and 3% of One Nation voters.
Where there is clear evidence of the methods change making a difference is in the don't-know response for Morrison's satisfaction rating. This dropped from 11% to 5%, the lowest neutral response for a Prime Minister since John Howard recorded 4% in the final week of the 2007 campaign. The only other scores so low were a string of 4% and 5% don't-know results for Bob Hawke in the 1987 campaign. So clearly the methods change has made some respondents who would previously have been undecided express a view, and it may just be that view tends to be negative. If so that would make it very hard to get a good rating, since Newspoll leader ratings were harsher than those of other pollsters already. We'll have to see over subsequent polls.
The methods change might also have impacted on the perennially skewed "Better Prime Minister" rating, on which Morrison "leads" Albanese 46-35, his smallest lead over Albanese to date. It's common for Opposition Leaders to gradually improve on this indicator as they become better known, so it is difficult to read anything into Albanese's modest three-point gain on this score.
How good or bad was the old Newspoll?
The previous incarnation of Newspoll was used at the following state and federal elections: 2016 federal, 2017 WA, 2017 Queensland, 2018 SA, 2018 Vic, 2019 NSW, 2019 federal. The table below shows the average errors for the primary votes and (where published) Coalition 2PP in the final Newspolls for each of these seven elections. A minus means the poll had the party lower than its result. Two averages are given - one for the absolute errors (how far out the poll was, irrespective of direction) and one to determine if the errors tended to skew to one side or the other.
(Correction: the Ave Error figure overall in the above table should be 1.79 points).
Overall in this time Newspoll displayed no significant skew for or against any party, on either a primary or 2PP basis, but it had some significant errors in the primaries at some elections. Four of the seven final polls were rather accurate (one of these extremely accurate) while the 2018 SA and Victorian polls and 2019 federal poll had significant primary vote errors. In the latter two cases this flowed through to significant 2PP errors, while in the SA election Newspoll did not release a 2PP estimate. Had it done so, the error would have been smaller in that case (maybe around 1.5 points) because both major parties were overestimated. The failure at the 2018 Victorian election was actually larger than at the 2019 federal election, but was largely ignored because the side predicted to win easily did so. On average, this incarnation of Newspoll was still a good poll by world standards, but Australian standards have been higher.
Will the new Newspoll be better?
Some of the announced properties of the new Newspoll are promising in terms of improved accuracy. For instance, the advertised "synthetic sampling procedures" should reduce the problems caused by having to scale up from small and perhaps unrepresentative samples in particular age groups, which then makes a poll more volatile because any error in these small samples has more impact. However (i) we still don't have enough detail about the poll's methods to be sure it's not doing anything new that might be unwise (ii) to the extent that the 2019 failure might have been caused by an unusual break among disengaged voters, it's doubtful whether anything will stop that happening from time to time in the future. Disengaged voters are unlikely to be found filling out online surveys in return for points that they might (if they get enough) be able to redeem eventually at a rate of something like $5/hour.
Hopefully in advance of the next federal election we will see these methods given some testing at state elections. Queensland (October 2020) and WA (March 2021) are scheduled to have state elections before the next federal poll, while SA (March 2022) could also be before. Tasmania (around March 2022 if parliament goes full-term) could be before too, but there was no Newspoll of the 2018 Tasmanian election. Online sampling poses extra issues with building an adequate sample for a state as small as Tasmania, and amid the general downturn in the level of state polling I'll be surprised if we see a 2022 Tasmanian Newspoll (although, of course, one lives in hope.)
However, as the results above show, a few state elections will not be enough evidence to conclude that the new poll is a safe bet in a close contest, and indeed it's unlikely that any poll could be such anymore. The previous version was outstanding at two of its first three elections, but not so good at three of the four after that.
No comments:
Post a Comment
The comment system is unreliable. If you cannot submit comments you can email me a comment (via email link in profile) - email must be entitled: Comment for publication, followed by the name of the article you wish to comment on. Comments are accepted in full or not at all. Comments will be published under the name the email is sent from unless an alias is clearly requested and stated. If you submit a comment which is not accepted within a few days you can also email me and I will check if it has been received.