Psephology And The Palace Letters

It's a nearly annual tradition on this site to release something every Christmas Day.  (Click the Xmas tag for previous examples.) It's also an annual tradition when I write the annual review, to list pieces that languished in the draft section, some barely started, others more or less complete but never quite seeing the light of day.  When I did this last year there was an unusual level of interest (a whole three enthusiastic requests!) in one of the unreleased items, about "psephological themes in the Palace Letters" and I decided to make it this year's Christmas present to readers.  This article mostly focuses on the 1974-5 cycle although the letters cover the 1975-7 cycle as well.   By the way, I am not the only psephologist dropping down chimneys with offerings this year.

The Palace Letters are a series of correspondence (not all of it related to federal politics) between the Queen's Private Secretary Sir Martin Charteris and Governor-General Sir John Kerr in the leadup to and the aftermath of the November 11 1975 sacking of the Whitlam Government.  They also include a large number of media articles that are public documents but would not be well known to a modern audience and that are fun to read over.  The letters were released after a very prolonged legal stoush, especially thanks to the persistence of Whitlam biographer and political historian Jenny Hocking.  

When Federal Polls Are 50-50, Oppositions Rarely Win

For nearly a year of federal polling, an unusual situation has existed.  The Prime Minister, Scott Morrison, has polled very high personal ratings, and large leads on Better Prime Minister scores, but the Coalition has struggled to build any meaningful two-party voting intention lead.  If polling is to be believed, an election held right now would be a close thing, though the government would probably survive.  

However, many observers just don't believe Labor is really at 50-50.  This is not surprising given that Newspoll was 3% out on the two-party preferred vote at the last election, so perhaps that's still the case and the government is 53-47 ahead and cruising.  (The history of polling failures elsewhere suggests probably not - every election cycle is different.)  But what I've noticed a fair bit is that people who don't believe the voting intention polling cite the PM's personal polling (or in some cases the primary votes) as evidence that the 2PP voting intention polling is wrong.   This doesn't make sense - why should voters give misleading answers on their current voting intention but not on what they think of the PM?  Or if sampling issues are causing the voting intention polling to be wrong, surely they would also drag the PM's personal ratings down (in which case he would actually be amazingly popular, and there would be a new issue of why he had only a modest 2PP lead instead of a massive one.)  

Oh No, This Wasn't Just An "Average Polling Error"

As previously noted, Australian opinion polling has just experienced its first clear predictive failure, in pick-the-winner terms, in a federal election since 1980.  Every campaign poll by four different pollsters (one of them polling under two different brands) had the Labor Opposition ahead of the Liberal-National Coalition (as it had been for the entire term), and yet the Coalition has won an outright majority.  Moreover, polls in the final weeks were extremely clustered, with 17 consecutive polls (plus an exit poll) landing in the 51% to 52% two-party preferred range after rounding, a result that is vanishingly unlikely by chance.  No pollster has yet made any remotely useful contribution to explaining this clustering - those who have even commented have generally said they didn't do it and it must have been somebody else.

The general reaction has been dismay at this unusual level of pollster error in a nation where national polls have a proud record of accuracy.  The Ninefax press, as I call them (SMH/The Age), have even announced that they now have no contract with their pollster, Ipsos, or with any other pollster.  (This may just be for show, since in the past Fairfax often took long breaks in polling after elections.)  News Corp is, for now, standing by Newspoll.  The Association of Market and Social Research Associations has announced a review, although this may be of little value as its only member who is involved is Ipsos. 

Poll Roundup: 2015 Year In Review

2PP Aggregate: 53.5 to Coalition (+0.1)
Coalition would win election held now with unchanged to slightly increased majority

It's just about the end of another year in federal polling; should any unexpected late polls appear I will edit this article to add them in.  After an update for this week's polling I'll launch into an annual review along similar lines to last year's.  From here on in the pollsters tend to go into summer recess with Morgan and Essential returning in mid-January and the heavy hitters coming back in late January and early February.

This week's polls

This week we have had readings from Morgan and Essential, which continue to sit at opposite ends of the Turnbull-era spectrum, this week returning 56-44 and 52-48 respectively.  The former was Morgan's highest reading for the Coalition this term, and the respondent-preferences reading was even higher (57.5%).  Essential has had the Labor primary at 35-36 in the last four weeks while Morgan has had it at 28.5 then 27.  Either both are wrong or one is very, very wrong.

Although both pollsters showed an uptick to the Coalition, this was tempered by the Ipsos from a few weeks ago falling out of sample, so the net result is just a 0.1 point gain, for the Coalition, after everything, to finish the year in exactly their 2013 election result position.  At least, that's my take; as usual recently, others may well be higher.  (Edit: Yep; Bludgertrack 54.1 Mark the Ballot 55 and Phantom Trend 55.2.  MtB assumes zero-sum and includes Morgan but not Essential, and Phantom Trend treats Morgan as having the same sorts of house effects it's had for decades, so those points explain why the latter two are so high.)

Poll Roundup: One Year On / The Magic Of Newspoll Rounding

2PP Aggregate: 51.7 to ALP (+0.1 since last week after no change in previous week)

The Government's First Year In Polling

This week's polling marks the end of the Abbott Government's first year in power.  In summary, the Government has spent most of that year behind in the polls.  Of the seven first-term governments for which polling exists, it is the first to spend most of its first year trailing in the 2PP vote, and only the second known to have been behind at all.  However, it ends its first year in arguably better shape than the Whitlam government did.  At the end of its first year the Whitlam government trailed 43-50 on primaries (equivalent to about 46:54 2PP) in a single Morgan poll, and while there was probably a bit of bouncing in that particular poll, it had been behind for a couple of months prior to that too.  Less than six months later, Whitlam's government went to an early election and was returned.

Early Abbott Era Polling Roundup

(Note: an update has been added at the bottom following the surprising first Nielsen result and other signs of narrowing in late November.  Another has been added for the release of official 2PP preferences.)

We are now just over two months into the lifespan of the first Abbott government and it is time for a brief summary of its polling history so far, and that of its leader. 

The following polls have been published since the 2013 election: nine Essentials (one of them including a week of pre-election data), four Morgans, two Newspolls and one and a half ReachTELs.  (The half ReachTEL was a poll of NSW and Victorian voters only, that canvassed voting intention along with Shorten-vs-Albanese ALP leader polling.)

Two-party preferred

Compiling a meaningful two-party preferred picture from this polling has not been easy, for the following reasons:

* The performance of Essential and Morgan in the period between the return of Kevin Rudd and the 2013 election.  Essential tracked in a contrary manner to other pollsters (below them for Labor after Rudd returned but then above them for Labor as the election drew closer).  Morgan (using preferences modelled by the last election) initially tracked favourably for Labor but once the election was called tracked neutrally.

* A method change by Morgan to drop online surveying from their most recent survey. 

Poll Roundup and Seat Betting Watch: New Dawn Fades Edition (August 27)

2PP Aggregate (Tuesday 27 August): 52.3 TO COALITION (+0.3 since last week)
Individual Seat Betting: Labor favourites in 59.5 seats (-1.5, Eden-Monaro, Brand becomes tossup)
Seat Total Market: Labor 61 seats (-2) (This figure is probably slightly skewed by longshot bias.)

This is week nine in a regular weekly series in the leadup to the federal election.  Week eight was here and through it you can click back to the previous weeks.  Or just click the "betting" label at the bottom.  As stated before, the aim of this exercise is not to claim that seat betting markets have predictive value, but to test whether they do, and to see which of the markets and the aggregated polls see the ultimate outcome of the election first.

This Week's Polls

It was a relatively quiet weekend for polling compared to the previous week's six-poll deluge.  Nielsen and Newspoll both returned 53-47 to the Coalition.  Roy Morgan returned 52.5-47.5 by last-election preferences.  Essential returned a second consecutive 50-50, but this one was off the back of a two-point fall in the ALP primary and a three-point rise in the Greens primary. My aggregate has gone to 52.3.  At the moment, it has one of the more benign readings for Labor doing the rounds - aggregates that benchmark solely off the last election are about a point worse, and those that handle sample size issues in a more sophisticated way, or weight heavily for accuracy of particular polls, are also showing slightly worse readings for Labor.  An aggregate of all the aggregators listed in Poll Bludger's excellent review of the state of things here comes out to 52.85.  My Newspoll rolling average, which I use heavily in historical modelling, is at 53.

Poll Roundup and Seat Betting Watch: Ghost of '75 Edition (20 August - Updates Added)

TASMANIAN POLLING COMING: Check here on Saturday and Sunday mornings from 9 am and there will be extensive commentary on state and federal results.  ReachTEL are in the field tonight and I was even polled! Questions are federal voting intention, preferred PM, issue importance, state voting intention, and rate the performance of the state government.

(Apologies to Tas Times readers looking for that stuff; I send the wrong link!  Correct link goes here)

-------------------------------------------------------------------------------------------------------------

2PP Aggregate (Tuesday 20 August): 52.0 TO COALITION (+0.5 since last week)
Individual Seat Betting: Labor favourites in 61 seats (-2, Forde and Moreton)
Seat Total Market: Labor 63 seats (-2) (This figure is probably slightly skewed by longshot bias.)

In this issue:
* Newspoll to a place no-one has won from.
* Is Essential just a Poll-Shaped Object?
* Robopolls: confessional for homophobes #
* Why Rudd Is Not The Electorate's Dumped Ex-Girlfriend
* Non-zero estimate of Labor's chances!
* Ghosts of 93 ... and 75.
* ALP not favourites in any Coalition seat.
* Betting Market Debate: Do Punters Just Dawdle Like Sheep?

Poll Roundup and Seat Betting Watch: Scruff Of The Neck Edition (August 13)

2PP Aggregate (Tuesday 13 August): 51.5 TO COALITION (+0.7 since last week)
Individual Seat Betting: Labor favourites in 63 seats (-2, Forde to Labor; Banks, Reid, Brisbane to Coalition)
Seat Total Market: Labor 65 seats (-2) (This figure may be slightly skewed by longshot bias; see below.)

This is week seven in a regular weekly series in the leadup to the federal election.  Week six was here and through it you can click back to the previous weeks.  Or just click the "betting" label at the bottom.  As stated before, the aim of this exercise is not to claim that seat betting markets have predictive value, but to test whether they do, and to see which of the markets and the aggregated polls see the ultimate outcome of the election first.

Things aren't looking too good for the Government after a few weeks of apparent competitiveness.  The polls say they're losing at the moment, the punters strongly think they'll lose, the voters think they'll lose and the modellers think they'll lose.   Still left expecting a Labor victory are really just the true believers, the Abbott-haters (and not even all of those), the Sawford formula devotees and Peter Brent.  There's a general perception that the Coalition now has this election by the scruff of the neck, that the first month of the Rudd comeback polling was partly bounce and that the government won't be going up from here and may indeed be going down further. Is it really quite that hopeless for Labor?

A Field Guide To Australian Opinion Pollsters

As of September 2016 this is an archived old edition that is no longer being updated.  For the 45th Parliament (2016-) edition go here.


Introduction

There are a lot of polls about in Australia these days.  But how do they all work, which ones have runs on the board and which ones can you trust the most? 

With an increased number of pollsters now polling in Australia, it's been suggested that an article describing what is known of each pollster and its strengths and weaknesses might be useful to many people.  So I thought I'd put one up, and edit it over time as the need arises.  (Now edited post-2013 election.  Last major edit 30 Nov 2013. Last minor edits 21 June 2016.)

The gold standard for success for an opinion pollster is that its polls at election time get the result as close to right as possible.  However, many of the new pollsters are little-tested against actual elections, and getting a specific election right is a combination of skill and luck.  In elections where there is a swing on the last day or two of the campaign, a pollster that is actually not polling correctly may have its errors cancelled out by the swing, and hence record a lucky hit.  There is more to being a good pollster than just getting it right at election time - a good pollster should also provide useful data between elections and do so using well-designed questions that are easy to interpret.  And a pollster should also present their data in a way that makes sense and isn't misleading or confusing.

Newspoll Upfront Exclusion Rates Since 2002

Advance Summary

1.  Some observers think that the proportion of voters who are indicated by Newspoll as "uncommitted" or "refused" gives Labor a much greater chance in the upcoming federal election, because the Government may attract a late swing from undecided voters.

2. The view that there is generally a late swing to an incumbent Government is false.

3. The view that there is usually a late swing to the party that is trailing is true, but it is not clear whether this is because there is a genuine "narrowing" effect, or whether this is because the Coalition tends to outperform its polling and the Coalition has often been the trailing party.

4. It is possible (and logical) that there is a link between increases in the Newspoll exclusion rate and poor polling for Labor, suggesting that when the party is performing poorly more of its voters become "uncommitted".  However, the evidence on this is inconclusive.

5. While it is plausible that undecided voters will move back to the Government and improve its result by, say, two points, this is also something that may not happen at all.

6. The issue of uncommitted and refused rates is only relevant to the margin of an overwhelmingly likely Coalition win (assuming no Labor leadership change).  There is nothing to suggest it is capable of turning the election.
------------------------------------------------------------------------------------------------------------------------

How Common Is A Five-Point Swing In Under Four Months?

Advance Summary

1. The Gillard government now probably needs a swing back in public opinion of around 5% in three and a half months to win the federal election. [Update:  After Monday's polls, it's even more than that.]

2. A recent article by Simon Jackman draws attention to swings of this magnitude, from one election to the next, being rare events that mainly (not exclusively!) accompany changes of power.

3. It is not correct to infer that swings of this size in a much shorter period are even less probable, because in fact 5-point swings within 16 weeks are commoner than 5-point swings between elections.

4. However, 5-point swings in shorter time periods typically result from temporary factors - leadership and event bounces, honeymoon effects, policy mistakes that are corrected, bad patches etc.  

5. Fast swings that have occurred in the leadup to elections have usually washed out of the system partly or entirely by election day.

6. The past history of quick 5-point swings in polling provides no sound basis for belief that even an extreme event such as the S11 attacks or leadership change would save the Government now.  (This does not mean it is impossible, just that there is no precedent.)

The Silly Season 2: End-Of-Year Poll Myths

Advance Summary

1. There has been some recent debate about the Government's polling position going into an election year.

2. The idea that this position has any special significance in projecting results is baseless.

3. While some pro-Labor sentiment has compared Labor's position now to Howard's end of year positions before elections that he won, in only one of three cases was Howard's position even arguably as bad.

4. Despite this there are earlier precedents for victory from seemingly quite poor end-of-year positions.

5. Analogies with Howard's position at the end of 2006 are fatally flawed because of the Labor leadership handover.

6. "Momentum" is a common concept in opinion poll commentary, but it has no basis in reality.  Movements in one direction from poll to poll are most often followed by movements in the other, probably mostly as a result of random bouncing from sample to sample.

7. Attempts to define a poor end-of-year position for Labor are already outdated, having been brought into question by more recent polls. (This article gives reasons for considering Morgan Face 2 Face to be valid data and not ignoring it completely.)

This article also contains an unrelated section examining an argument by Peter Brent that a given level of voting intention is more durable for an unpopular leader than a popular one.  There is not enough evidence to apply this to unpopular opposition leaders, and especially not to Tony Abbott at this time.

--------------------------------------------------------------------------------------------------------------------

In a previous article (Is The Silly Season Real?) I examined the idea that opinion polls behave strangely in December as many people swing into Santa mode and found that, at least in the case of Newspoll, there is no evidence to support it at all.

Another form of  commentary that I have seen quite a lot of lately involves focus on the government's polling position at the end of the year and how this compares to that of previous governments, and what this might (supposedly) tell us about whether the government can win.

As Gillard recovered, so can Abbott?

Advance summary:

1. The view has been advanced that since Julia Gillard has become much less unpopular, then so can Tony Abbott.

2. This view does not automatically follow, because Gillard and Abbott occupy different positions and became unpopular for mostly different reasons.

3. The record since the late 1960s shows that the average recovery in net satisfaction for both Prime Ministers and Opposition Leaders between their worst rating and the next election is about 20 points.

4. However, very unpopular Opposition Leaders are usually removed, and those who make it to elections have so far lost them even if they had ceased to be unpopular by election time.

5. There is one case of an Opposition Leader recovering from a rating worse than Abbott's to a positive net rating within two months, but without winning the election.  (Andrew Peacock, 1984)

6. However, the general historical trend is against the likelihood of Tony Abbott recovering to a positive or even near-zero rating before the next election.

Leadership Transition, Polling And The Greens

 NOTE: This is an old article that has had various minor data updates and a short postscript added in May 2013.  The basic conclusions are not affected.

Advance summary of this article

1. Recent election and polling results show the Australian Greens are clearly struggling.

2. Furthermore, the new leader, Senator Christine Milne, has polled poor personal approval figures.

3. Attempts to connect the slump in Green fortunes with leadership transition are inconclusive for many reasons. 

4. Nonetheless, the change of leadership has at least not stopped a flow of votes from the Greens back to Labor. 

5. The current situation is consistent with Milne's track record as a poor electoral performer when leader of the Tasmanian Greens in the 1990s.