Senate Reform Performance Review Part 2

In this piece I continue the roundup of the performance of the new Senate voting system that I started in part 1.  Part 1 covered proportionality, half-Senate projections, winning vote shares, the impact of preferencing, just-vote-1 rates and exhaust.  This part covers  informal votes, the One Nation question, below the line rates, the two most unusual victories (Lisa Singh and Malcolm Roberts), the impact of the fixed quota, the question of verification and areas for improvement. (There's also a brief note about Inclusive Gregory that isn't relevant to reform, but that I thought I should mention anyway).  Anything else I think of may be tacked onto the end.  I have already discussed donkey-voting, proximity preferencing, the abject failure of most parties' how-to-vote cards, and preferencing patterns by state on the Button Press Week thread. Also, lots of Tasmanian-specific stats can be found on the Tasmanian button press thread.



Informal Senate Votes

The Senate informal vote rate at this election increased from 2.96% to 3.94%.  Given the confusion created when any new system is introduced, this is not a very large increase, but I still believe it's a bit larger than it should have been.  That said, the rate was slightly lower than the rate at the first two elections under the previous Group Ticket Voting system (1984 and 1987).  It also hardly stands out as an outlier compared to previous years, being about the same as at three of the last five elections.

(Before the 1984 changes, informal rates were usually scandalously high because voters had to fill in all the boxes.)

Here is a graph of the changes by electorate (noting that some electorates were heavily redistributed):

The top five electorates for informal voting (position in 2013 in brackets) were Blaxland (1), Watson (2), Fowler (3), McMahon (5) and Werriwa (10). Calwell (Vic) fell from fourth to sixth.  The top three are also the same top three as in 2010, but in 2007 the top five electorates were Victorian.  Now they are all in New South Wales.

Informal Senate voting fell in five electorates, generally inner-city seats: Grayndler, Melbourne, Reid, Higgins and Perth.  The six greatest proportional increases, however, were all in Queensland (Fairfax, Fisher, Wide Bay, Moncrieff, Fadden, McPherson), and Queensland had ten of the top eleven (the odd one out being Indi.)  If you spotted the (not necessarily causal) pattern that the Senate informal vote went up lots in seats Palmer United Party did really well in in 2013, you're not alone.

Given the dire forecasts about how the AEC would not have time to educate voters on how to use the new system properly, the increase is remarkably modest.  All the same I can think of two things that are worth improving to reduce informal votes for next time:

1. The savings provision for below-the-line votes, which requires voters to have the numbers 1 to 6 once and once each, is too stringent.  Voters sometimes accidentally omit or repeat numbers in this range and shouldn't have their whole vote junked for doing so.  An improvement would be to allow any below-the-line vote containing the number 1 once, provided that it had at least 12 boxes numbered with at most two sequence errors in the first 12.

2. Too many voters are voting both above and below the line, causing their vote to either be informal or else potentially prematurely exhaust (more often the latter if they start above the line).  Even if retained as formal some such votes have little relation to what the voter was trying to do.  The ballot paper needs to have a clear and prominent instruction to "Vote above or below the line, not both"

Who Put All Those Hansons In The Senate?

Debate about the role of Senate reform in the election of multiple One Nation Senators has been raging since election night.  While Labor is blaming reform for the One Nation victories, during the 2016 reform debate several Labor MPs actually said that only the Coalition, Labor, the Greens and NXT would be winning seats under the new system.  Yet not only has a minor party outside that list won four seats, but three of the previous non-Green crossbenchers have been returned and Derryn Hinch has joined them.  So much for Labor's claims that the new system would "disenfranchise" a quarter of Australian voters (and there are many more examples of such claims).

Some people did actually claim Senate reform would advantage One Nation before the election (eg Saturday Paper, The Australian, Van Badham).  The argument was that in the absence of parties preference-trading against them (as happened in 1998-2001) all One Nation needed to do was poll a modest primary vote and then they wouldn't need to reach a full quota since the exhaust rate would save them from being chased down by others on preferences.

I pointed out the flaw in the comparison to the old system before the election. Had the 2013 election been a double-dissolution, Pauline Hanson would have won a seat in NSW off 1.22% of the primary vote (a result her party easily exceeded in every state this time around, including in Queensland as surplus over their first quota).  The image of One Nation as a party that was easy to beat under Group Ticket Voting hailed from a very different era in which nearly all votes not for One Nation were for the Coalition, Labor, the Greens and Democrats, making fencing One Nation out on preferences easy.  Since then we have seen a massive rise in the micro-party vote, and not just similar right-wing micros but also some centrist and left-wing ones were sending ATL votes to One Nation.

As for exhaust, results as expected showed that a 3%-ish vote is no guarantee of winning without strong preference flows.  But it turned out that one of One Nation's wins was on raw quota and in the other three cases the party was heavily outperforming parties that might have beaten it to the final seat (overtaking five of them in Queensland).   Moreover, One Nation very nearly crossed quota before the seats were all decided in NSW and WA.  Indeed, far from exhaust stopping One Nation from being caught on preferences as the reform-helps-One-Nation theory claimed, One Nation missed out on a very real chance of beating the Greens in Tasmania because Liberal Party votes exhausted!

By any fair measure, One Nation's primary vote share and strong preferencing performance compared to other micro parties entitles it to the seats it won.  Anyone who disagrees with that has a problem not with the new Senate system, but with preferential and proportional democracy itself.  All the same, the fact that we have four new One Nation Senators and not one (given the votes actually cast) is down not to Senate reform but to the government's strategic decision to force a double-dissolution.  Here's a contingency table of the possible outcomes for One Nation based on the votes cast, depending on the kind of election and whether or not there was Senate reform:

Without knowing what deals might have been done we cannot know for sure how many seats One Nation would have won under the old system - it might even be in theory that at a half-Senate election Hanson herself could have been beaten by some similar right-wing micro-party on a preference snowball.  It's very likely though that PHON would have won at least one at a half-Senate election with a good chance of more, and that they would have won multiple seats at a double dissolution.  Depending on the deals done, their seat haul at a double dissolution under the old system could have even been as high as seven.

The big question is how a discredited relic party that couldn't even get its GVT form in in Victoria in 2013 managed to multiply its vote by ten in South Australia and nearly 17 in Queensland.  My very first assessment of One Nation's chances was way too negative precisely because of their dreadful 2013 result. Some factors in their revival include the demise of Palmer United, a resurgence of anti-Islamic xenophobia, and the publicity given to One Nation by the media on the back of the reform-helps-One-Nation theory (which may have thus become a self-fulfilling prophecy in spite of being false.)  But others over which the left had plenty of control included the nature of the campaign (all very urban, middle-class and modern, nothing to see for the old right or the left-behinds from either major party) and also conspiracy theories about Senate reform.

By opposing reform Labor gave oxygen to claims about its impacts that until then had been the domain of conspiracy cranks and vested interests in the preference-harvesting industry.  The results have shown that these theories had no business in the political mainstream, but Labor insisted on putting them there.  If the increased vote for micros including One Nation (a magnet for conspiracy nuts, after all) was in any way a backlash against Senate reform, then Labor fuelled that backlash.   This puts Bill Shorten in a shaky position from which to blame anyone else.

Below The Line Vote Rates

Below-the-line voting increased to the highest level for at least the last five elections in every state, falling only in the ACT:

The increase was most marked in Tasmania (where BTL rates nearly trebled because of a below-the-line rebellion against party preselections) and New South Wales (where the rate was easily more than double that for the past four elections).

The increase in below-the-line voting resulted in Labor winning a seat in Tasmania that would have otherwise gone to One Nation (see Lisa Singh section below), and caused a different Labor Senator to win than the candidate in line for a seat by ticket order.  That seems to have been the only case of below-the-line voting actually changing a result, but it does show potential for this to someday happen in other states.  Although below-the-line leakage from the Labor ticket in South Australia was greater than Bob Day's victory margin, the proportion of leakage caused by increased below-the-line rates was not.

Lisa Singh's Historic Below-The-Line Win

Lisa Singh's win of a seat from sixth place on the Labor ticket marked the first time a candidate had won a seat out of ticket order since 1955 (at which time party boxes did not exist.)  In the early years of the PR system introduced by the Chifley government, there were a few cases of Tasmanian candidates winning out of party order, usually because their parties had put popular candidates low on the list to try to win more seats.  The 1953 case of Bill Aylett (Labor) was comparable to Lisa Singh's - Aylett was initially deselected, then reendorsed for the bottom position following a death.

Singh's win with 6.12% of the Tasmanian primary vote was achieved entirely on below-the-line primaries and preferences with not a single party vote reaching her.  Under the old system the onerous nature of filling out 58 boxes would have given her no chance of re-election.  The strange thing though is that Labor accidentally benefited by demoting Singh to the so-called "unwinnable" position.  Singh drew votes to the Labor ticket that Labor would never have otherwise received, especially from the Greens.  Without these votes Labor would have had little more than four quotas and would not have won five seats.  The Liberals also could have pulled off the same thing for Richard Colbeck, had their share of all preferences in Tasmania not been so poor.

Singh's victory shows that it was well worth liberalising below-the-line voting as it has given voters an avenue of appeal against party preselections, while still allowing those who want to vote the "party card" to do so easily.   To what extent Singh's win was also caused by the reforms to above the line voting isn't clear.  It might seem that she would also have won off her quota of BTL preferences had only BTL been liberalised (creating a system like the Victorian upper house), but in that case parties from which Singh obtained BTL preferences in the real count may have snowballed ahead of her on ATL group ticket flows.

While this may all seem a specifically Tasmanian thing (because of Tasmania's familiarity with multi-candidate voting) it might someday work in some other state.

Malcolm Roberts And Those 77 Votes

Malcolm Roberts' win from a small One Nation surplus, with a personal primary vote of just 77, has created a lot of misinformed commentary.

Senate reform as such has nothing at all to do with Roberts' victory with such a small BTL vote.  At the only previous double-dissolution under the group ticket system (1987) many major party Senators were elected with very small below-the-line votes - the previously record low 155 for Noel Crichton-Browne (Lib), 168 for Robert Ray (ALP), 208 for Amanda Vanstone (Lib), 201 for Chris Schacht (ALP) and so on.  Had this election been a double dissolution under the old system, it's entirely possible Roberts would have won off exactly the same BTL vote.  He won because voters for other parties directed huge numbers of mainly above-the-line preferences to One Nation.

The main reason for Roberts' new low is that this is the first case we have of a minor party that does not have unusually high BTL rates winning two seats, since party boxes were introduced.

I am not aware of anyone predicting before the election that One Nation might win two seats in Queensland or even mentioning it as a serious chance.  This might be because people thought that if One Nation did reach quota it would be with a very small surplus, and hadn't realised that under the new system a very small surplus can be enough to beat micro-parties.  (This was really the biggest new thing we learned from the results, except that the rates of following how-to-vote cards are much lower than would have been expected.)  The other strange thing with Roberts is that while he was well known before the election as a "climate change denier", revelations of Roberts using "sovereign citizen" language in protest against the carbon tax did not surface in public debate until after the election.

Perhaps it would have made no difference, but there is a question of whether opponents of parties like One Nation are researching their candidates enough in advance to spread public awareness of possible evidence that One Nation could be as crazy a rabble as ever.  In the 2014 Tasmanian state election, the Liberal Party used detailed candidate research (some of it mine, conducted independently of theirs) into the wackiness of the state Palmer United crew in a manner that helped burst a PUP polling bubble and prevent PUP winning any seats in the state.  I also wrote a candidate guide that pointed out some of the pitfalls lurking on the 2016 Tasmanian Senate ballot paper, which may have helped damage Family First's preference flow in the state.  When I look at the reaction to Roberts' eccentricities, it all seems surprisingly post-hoc.

In terms of the mechanics of Roberts' victory, he was competing with other micro-parties (eg Liberal Democrats, Animal Justice, Katters Australian Party, Nick Xenophon Team, Family First and Glenn Lazarus) for the preferences of fellow micro-parties.  The preferences of Labor, Liberal and Green (the parties with the highest how-to-vote card follow rates) were largely irrelevant, and since hardly any voters followed the micro-party cards, they were making their own decisions.

Of these micro-parties, One Nation was the most preferenced among voters for (among others) KAP, GLT, Shooters Fishers and Farmers, and Australian Liberty Alliance - fairly predictable right-wing sources.  But it was also the most preferenced by voters for some of the left micros (Sex/HEMP and Drug Law Reform) and the second most preferenced by NXT voters.  KAP, ALA and Derryn Hinch Justice Party were especially rich preference sources for One Nation. The party's huge profile advantage over the other micros helped it gain massively on preferences and easily haul in the Liberal Democrats (whose preference performance was poor).

Fixed Quota

An issue with the new system as implemented is that vote-values are being locked up unnecessarily in party quotas as votes exhaust out of the system.  In Tasmania, after the exclusion of Richard Colbeck, 6493 votes had exhausted, meaning that any candidate with 24790 votes or more would certainly be elected (the quota was 26090). Lisa Singh had 26094 votes at this point, so was 1304 votes clear of what she needed to certainly win, yet her surplus was only four votes.  Likewise, Liberal candidate David Bushby was 10,291 over quota at this point, but 11,591 over the point at which his election was mathematically certain.

Had there been a progressively reducing quota (as votes exhaust) then Bushby and Singh would have had 1300 votes more apiece at this stage.  The net impact would have been that Bushby, Singh and Bilyk would all have had larger surpluses, and in total this would have caused McKim to win the final seat by more than he did.  So a progressively reducing quota wouldn't have changed the Tasmanian result, but the lack of it came very close to causing One Nation to win unfairly.  Also, even before Colbeck was excluded, there were thousands of vote-values locked up in the quotas for the eight already-elected Senators over and above what those Senators now needed to be sure of election.  An iterative quota, such as in the Meek System, would have freed these vote-values, although as far as I can tell Richard Colbeck would still have been excluded at the same point.

In South Australia, Labor's Anne McEwen was disadvantaged by the fixed quota. She had to compete with the Greens and NXT for left and centrist preferences as they took longer than necessary to be elected, and the top three Labor Senators had vote-values locked up in their quotas beyond what they needed.  As far as I can tell, a progressively reducing quota wouldn't have changed this result, but an iterative quota would have made the final seat so close that I'm not sure which way it would have gone.

There is a fair amount of fear of introducing an iterative quota in Australia because of lack of prior experience of it and impossibility of replication by hand.  However we should at least have a progressively reducing quota to stop some voter intention being wasted.

Magic Expanding Lambie Votes

Nothing to do with Senate reform, but it's worth noting that a very silly example of Inclusive Gregory distortion happened at this election, with some votes cast for the Jacqui Lambie Network in Tasmania increasing in value during the count.  Dean Ashley has found that votes the Liberals received at value 0.064 on the exclusion of Steven Martin were later redistributed at a value of 0.092 as part of the Bushby surplus (when in an ideal system they would have been reduced in value).  The impact on the count was tiny (as the papers were only worth 90 votes total) but this sort of thing still shouldn't be happening.

Verification

Some concerns about verification of the results have been raised by some electoral IT experts and retired electoral administrators.  The AEC outputs files that include details of every formal vote that is included in the count, making it possible for those with formidable IT skills to try to replicate the result.  Grahame Bowland has been testing the preference distributions based on these files and has provisionally found that they match exactly; similar replication attempts succeeded in 2013.

What cannot be so readily verified by the general public is that the AEC's files match what is actually on the ballot papers and that those votes classified as informal were in fact informal.

I spent about 15 hours scrutineering the Tasmanian Senate count and watched the process.  Scanned images of votes came through to human data-processors who would enter in the number they could see on the screen for boxes picked up by the system as containing numbers.  If something strange turned up the operator would look at the whole ballot paper as scanned.  If there was a mismatch between the scan and the human interpretation it was referred to another operator, and there were various chains of command for "raising" difficult votes at a higher level.  There was a separate computer desk where all the really tough cases went, which was easy for scrutineers to watch over.

In theory, the role of scrutineers in the system allows them to watch the human interpretation of votes and challenge any they disagree with.  In practice, this is only practical to a small extent.  By the end of the Tasmanian count the operators had become blisteringly fast, completing manual entry in what I calculated as an average of six seconds per ballot paper - many would disappear from the screen before a scrutineer could see the full preference flow.  The processing of so many votes in a room at once would mean that for every party to watch every vote getting processed, the room would be full of hundreds of scrutineers.  In any case most parties lack the resources to provide more than a handful of scrutineers for the post-count.

What scrutineers can do that is useful under this system is to watch the processing of informal votes, since many borderline calls are likely to crop up there.  They can also keep an eye out for any obvious issues in data entry that might occur on a regular basis.  But they simply can't see everything, and I do agree with the proposal that the matching of ballot papers to recorded preferences (both formal and informal) should be audited to check that preferences were recorded correctly.  For all I know, such a review may already be planned.

Overall the conduct of this election was remarkably smooth compared to pre-election concerns, and even compared to the previous election.

Areas for improvement

The following are some areas I think should be improved prior to future runnings of this system. I may add others.

1. The ballot papers should more prominently indicate that the voter is to vote above or below the line and not both.

2. Savings provisions for below the line votes should be improved to preserve more voter intention.

3. Rotation of parties on ballot papers should be seriously investigated to prevent proximity-preferencing and donkey-vote impacts from affecting close results, at least in double dissolutions.

4. A progressively reducing quota should be introduced.

5. Display of Senate results in the Virtual Tally Room should be improved so that there is one display of votes for each state that only includes those booths that have been fully processed as party votes and votes for specific individual candidates.  (Following and explaining the Tasmanian count was a nightmare because of the lack of such a feature and the releasing of misleadingly low interim  percentages for individual candidates.)

6. Inclusive Gregory for surplus distributions must be replaced by Weighted Inclusive Gregory, or with some other system that does not cause its distortions.

7. New South Wales should, if resources permit, amend its Upper House system to better encourage the use of preferences.

Some voters have asked me for a view on scrapping ATL voting boxes altogether and just going to candidate-only voting.  Based on the record of the system that ran from 1949 to 1983 (which was exactly that, but with compulsory preferences) it seems that cases of voters overturning party preselections were relatively rare - this happened in Tasmania at the first few elections, but not after that.  That was probably a consequence of single ticket orders within each party, so voters tended to vote down the list.  To get something more radical going in terms of individual candidates, it would be necessary to not only scrap above-the-line voting but also introduce Robson Rotation within party columns.  Ideally I think it would be a good thing if Senators had to work hard for preference shares and campaign individually but I think there's a long discussion ahead of us in terms of practicality and voter education before we could get to that point.  So that is not on my shopping list at the moment.

However major parties that think such a system would be bad for them should probably think again.  As we saw with Singh and could have seen with Colbeck, splitting up a party's vote between candidates rather than having it flow down the ticket creates extra paths to possible wins for the big parties.

How To Vote Cards

How to vote cards (and another free plug for David Barry's magnificent efforts in making ATL preference flows explorable) were generally very unsuccessful at controlling voter preferencing choices at this election.

There seems to have been a Palmer United Party how to vote card in Victoria that achieved a 51.6% follow rate (1-6 numbered in exact order), although I haven't seen it.  If that is so, it was the highest in the nation.  Apart from that the highest recorded anywhere was 39.7% for the Coalition in Victoria, and the Coalition averaged about 30% per state in the mainland states.  Labor averaged about 14% and the Greens perhaps 10%; multiple cards make it difficult to estimate the Greens figure correctly.  No other party averaged anything like 10% though some right-wing parties (Rise Up Australia, Australian Christians, Australian Liberty Alliance and Family First, for instance) got around that rate in some states.  Rates for left-wing micros varied from a few percent to fractions of a percent and in one memorable case zero.

A few commenters have suggested that just looking at the strict 1-6 follow rate underestimates the success of cards because they may have been influential on voters without the voters necessarily following them fully.  There's bound to be something in this, but I think it's easy to overestimate it, for the following reasons:

* Where left-wing parties preferenced like-minded but obscure micros second, it wasn't just that voters didn't get to the end of the card; the vast majority didn't even follow it to 2.

* A similar thing was observed with attempts by the Shooters, Fishers and Farmers to preference religious right parties.

* Even for the stronger flows, the gaps between the 1-2 flows between the top two parties and the full 1-6 card flow weren't massive, and could have been largely caused by normal voter intention rather than the influence of cards.

* Once a voter followed a card 1-4, they strongly tended to follow 5 and 6 as well; only a small proportion of voters went their own way from that point.

This is not to say that cards have no impact, and indeed had the Liberals in South Australia not preferenced Bob Day on their card he would probably not have won.  But it does seem that micro-parties that lack the ability to put boots on the ground outside booths also have little ability to impact upon preference flows.  Also, left-leaning voters especially like to make their own decisions.

This may all be partly a response to the novelty of the system and it may be that over future elections we will see HTV cards become more widely followed.

Note: Part 1 has been updated with a proportionality comparison with the 1987 DD.