Tuesday, September 24, 2013

Tasmanian Senate Seat Goes To The Button - Lambie (PUP) Wins

(Admin note to commenters: I have just done a cleanout of duplicate comments (where people submitted multiple copies of a comment).  If you get an email saying your comment was deleted, please don't feel rejected! One comment from a few months back was rejected for being off-topic but there will be a thread on the matter it relates to soon.)
==============================================

This is the thread where I will post the result and details of the preference distribution that determines the last Tasmanian Senate seat.  The thread that has been following the late Senate count and attempting to model the outcome has been extremely popular but has also become far too long and I want to keep the analysis of (i) the known result, once it is known (ii) any recount action that might be necessary, separate.

My thanks to the AEC for their very quick provision of preference distribution information.

Preview

On Wednesday at about 10 am the AEC's computer system will be commanded to perform a complete distribution of preferences for the Tasmanian Senate.  Liberal Senators Richard Colbeck and David Bushby, Labor Senators Carol Brown and Catryna Bilyk and Greens Senator Peter Whish-Wilson will be re-elected.  Unless all modellers modelling this count have somehow missed something surprising and important, Labor Senator Lin Thorp will lose her seat, which will go to one of Palmer United Party candidate Jacqui Lambie, Liberal candidate Sally Chandler or Sex Party candidate Robbie Swan.



Tasmania has had some close finishes in previous Senate counts (a few thousand votes difference for the final spot or a critical exclusion is fairly common) but this could well be the closest for a very long time and it has been the most complex to attempt to model.  You can see my various meanderings about this result on the thread linked to above.  We are in the remarkable situation that as we go "to the button" we simply do not know in advance what the outcome will be; we cannot even have much confidence which of the candidates will win.  To have that confidence would have involved having someone who knew exactly what they were doing giving up the last few weeks of their time standing over the data entry of a very large number of below-the-line votes.

The following, I think, are the consensus views of the modellers who have attempted to model this count, both on this site and elsewhere:

* The first critical point in the fight for the last place comes when there are seven parties (Greens, Liberal, LDP, Labor, PUP, Sex Party, Family First) remaining.  The Sex Party is extremely likely (some modellers say certain) to stay ahead of Family First at this point, in which case Family First are eliminated.  If Family First somehow avoid elimination at this point, they probably win.

* The second critical point comes after the exclusion of Family First, with six parties remaining (Greens, Liberal, LDP, Labor, PUP, Sex Party).  Whichever of the Sex Party and Labor is last at this point is excluded.  The Sex Party is notionally ahead "on the calculator" by 795 votes, but in practice needs to gain an amount variously estimated as between 280 and 605 votes over Labor from the below-the-line votes of a large number of minor parties (the exact target figure depending on the rate of leakage in the Labor ticket, which is hard to predict.)  No modeller thinks this is more likely than not to happen.  Some modellers think it is almost or even perhaps about as likely to happen as not to happen, while some think it is unlikely. 

* The third critical point comes assuming the Sex Party is excluded, with three parties remaining (Liberal, LDP, PUP) after the exclusion of Labor's Lin Thorp and the election of Peter Whish-Wilson.  If PUP are last at this point the Liberals win, otherwise PUP win.  The contest for last place is between PUP and the Liberal Democrats, who cannot win, but can spoil PUP.  The Liberal Democrats are notionally ahead "on the calculator" by 1469 votes, but in practice need to outperform PUP on the preferences of minor parties to exclude them at this point.  All modellers think PUP is more likely than not to survive this exclusion but no-one is confident in calling it.  It would probably be callable if not for the confusion caused by the name "Liberal Democrats" and the possibility that as the count unfolds they may gain preferences from voters filling out a number of squares and "donkeying" the rest.

It appears that PUP is the best placed of the three parties, but because they need both these last two critical points to go their way and there is a reasonable chance of failure on either, they should not be feeling especially confident.  Margins of perhaps only hundreds of votes are involved in a count with tens of thousands of below-the-line ballots.  The other two parties cannot be ruled out.

It will take a little while after 10 am for news of the result to filter through. 

10:10 am Announcement of the result is expected within about 15 minutes. 

Outcome

10:12 am Examiner journalist Ben McKay tweets that there are unconfirmed reports that Lambie (PUP) has provisionally won. Update: Now stated as confirmed by scrutineers.

10:33 am It is now confirmed by the AEC that Jacqui Lambie is provisional winner.  Awaiting distribution of preferences to see how close it was and whether there is risk of challenge. 

I have the distribution of preferences and the key margins were:

* Sex Party over Family First by 821.
* Labor over Sex Party by 244 (so close!)
* PUP over Liberal Democrats by a cushy 1276 .  Less close than expected but I did warn on the modelling thread that even with a calculator lead of over 2000 the LDP would not necessarily make it (if they had that).

And the moral of this story is that the ABC Calculator does not tell you the result!  Lambie had to overturn two calculator leads (Sex over Labor and LDP over PUP) to get this and did so.  

The margin is sufficient that I doubt there will be anything further by way of challenges (etc).

Analysis

The key margins are noted above.  The distribution is available here.  The result was the least unexpected of the three that seemed seriously possible, but I hope people can see from the closeness of one of the exclusions why there was great reluctance about calling it.  It appears to be the closest Tasmanian result (at a key exclusion) since the current Senate system started in 1949 (note that this system was modified by the inclusion of ATL voting in 1984).  Notes concerning issues canvassed in modelling will be progressively added in this space through the day.

As the Sex Party / ALP exclusion point was the closest I'll look at how that unfolded:

* Leakage from the #1 and #2 Labor tickets was relatively light compared to many projections with the loss of 124 votes from Carol Brown's surplus and 19 votes from Catryna Bilyk's.  The hold back from John Dowling to Thorp was 88.4%, slightly but not significantly below normal.

* Net loss from the ALP ticket to the Sex Party ticket at the Labor surpluses and exclusions was 292 votes.  However John Dowling gained 49 votes during the count from other parties so the net effect of ALP leakage was a loss of 243.

* The Sex Party made large BTL gains on Labor at three exclusions: the Pirate Party (Sex Party gains 155), the HEMP Party (Sex Party gains 71) and perhaps surprisingly the Shooters and Fishers Party (Sex Party gains 80).  On the latter, see the excellent modelling by Christopher Burge near the end of comments to the modelling thread here in which he used an old Victorian count to project that the Sex Party could gain 70.  It seems that Shooters and Fishers voters do not see their party as a right micro and the Sex Party as a left micro but rather see both as having a compatible degree of pro-freedom attitude.

* Despite this, if we add up all the exclusions and surpluses on which the Sex Party gained, they still fell over 60 votes short of catching Labor on those.  A part of the problem here is that some expected gain exclusions, like Australian Independents, were slight gains to Labor, while others, like Stable Population Party, were not significant gains to the Sex Party.

* Labor's margin over the Sex Party was further bolstered - if you can call 244 votes that - by a string of minor counts where the ALP made small gains.  These most prominently included Liberal Party leakage (44 votes), leakage from Green candidate Helen Burnet (37 votes), the Country Alliance (29 votes), and the DLP (26 votes) as well as a long string of others.  Ticket order would have influenced the Country Alliance flow and gender voting and Thorp's remaining left-cred the Greens flow, but the general impression I got was of Thorp picking up preferences here, there and everywhere because she had profile in Tasmania while Swan did not.

It is possible that the Sex Party missed out only because they did not have reason to believe Tasmania was winnable for them (and nor should any "left" micro have believed they could win in the state but for the huge PUP vote providing a tasty bonus to anyone who could deal well enough and then catch them.)  If the Sex Party candidate for Tasmania had been an even vaguely well-known Tasmanian, I believe that they would have won this count from the starting position.

I will look at the LDP-PUP point and what happened there later.  The high margin - which was always possible - makes that one less interesting but it will be useful to see why it was so high.

 Actually, rather than me reinventing the wheel, for that one I suggest you have a look at Antony Green's coverage of calculator-to-actual differences here.

A comment about the ABC Senate calculator.  The calculator was indeed a useful tool for following and modelling the count but it again caused unnecessary trouble.  The trouble was caused by presentation which was not prominently clear about the calculator's limitations.  The disclaimers appearing in small print at the bottom should have appeared at the top of the page in, oh, at least 20-point.  That would hopefully have discouraged some of the bad reporting we saw of the post-count.

There are some really neat colour graphs by Truth Seeker of the BTL flows here.

Absurdities

In the end, the Tasmanian outcome is not that odd.  The party with 2.63 quotas won two seats, the party with 2.30 quotas won two seats, the party with 0.82 quotas won a seat and the party with 0.46 quotas beat the party with 2.63 to the last seat on preferences.  Could happen in Hare-Clark even with semi-optional preferencing.

But the process by which this unfolded is bizarre and I think it is worth highlighting some of the strange aspects of it to draw attention to the absurdities of Senate preferencing and the need for reform of a farcical voting and counting system:

1. The Liberals would have won had more voters confused the Liberal Democrats for them.  This would have resulted in the LDP eliminating PUP.

2. Liberal scrutineers had a vested interest in campaigning for the inclusion of votes for their traditional rivals the ALP in order to exclude the Sex Party and keep their own candidate in the race.  They also had an incentive to have their own votes ruled informal in order to reduce the quota for the same reason.

3. The Sex Party almost won.  Had they won it would have resulted in the election of someone from outside the state (but who I suspect would have been welcomed here by many) off a tiny percentage of the primary vote, and a 4-2 left-right split in a state in which Lower House seats split 3-2 to the right (if you count Wilkie as left, which I do).

4. Had PUP received more votes instead of those votes being informal, they may have lost.  The reason for this (see geekLections) is that more PUP votes would have increased the quota and reduced Labor's surplus, causing the Sex Party to eliminate Labor.  (I think that with an extra 844 votes PUP might have nipped the Sex Party at the three-party exclusion point with the Liberals, but that losing by having too many votes is even possible is rather funny.)

5. A crucial element of PUP's victory was that while nearly all the other micros were snowballing to the Liberal Democrats, PUP received one significant micro preference source: the Australian Fishing and Lifestyle Party (664 votes ATL).  Had these also gone to the LDP, PUP would have lost by twelve votes.

6. Two major contests in the count involved two parties of which one could win the seat and the other was running dead with no prospect of election. One of these was only so high because of a combination of voters mistaking the party's name and preference harvesting.  Its candidate had no profile in the state as shown by a negligible BTL vote.

7. The Family First party was only narrowly eliminated.  It was eliminated because the Sex Party vote surged on the day compared to pre-poll and postal voting and did so by more than the national average.  Had it only surged by the national average, Family First would have won.  Whatever the reason for this (and I'll not be shy about accepting a small part of the possible credit, though this might have more to do with it as the postal-to-day swing was strongest in the north) a victory for an extremely anti-gay Family First candidate on the preferences of Bob Brown's former party was averted by 0.25% of the overall Tasmanian vote.

8. Then there's the Inclusive Gregory system, an egregious violation of "one vote one value" that - at certain stages - causes the votes of leakers from the party tickets to lose value while party ticket votes increase in value or lose less value than leakers.  This advantaged PUP repeatedly in their fight to get over the Liberal Democrats, eg by downweighting leaks from the Labor and Greens tickets.

9. And finally - and this one I think is the most telling of all - is what happened to the Pirate Party above-the-line votes.  The Pirates had 1442 above-the-line votes, which went directly to the Greens.  By doing so, they increased the Greens' surplus by 1442 votes, compared to if they had gone somewhere else.  But the Green surplus included enormous numbers of Labor and Green above the line votes.  Because the Inclusive Gregory system weights surpluses by ballot paper numbers irrespective of their previous reductions in value, this meant that every Pirate ATL vote that reached the Greens was worth almost nothing from the Whish-Wilson surplus on, but increased the net value of the combined Labor and Green ATLs in the surplus by nearly a vote!  The result - Pirate Party ATLs that preferenced the Liberals ahead of PUP, causing a benefit to PUP over the Liberals ... of almost exactly the size by which PUP escaped elimination by the LDP.

But if you think this is bad, it is nothing compared to Western Australia, where the fate of as many as five different possible winners of the last two seats has been hanging on minute differences in whether the Australian Voice Party plus the Sports Party exceeds the Rise Up Australia Party, or whether the Shooters catch Australian Christians, or vice versa, or who prevails in a heavyweight showdown between Animal Justice and HEMP (neither of whom can win).  Worse still in NSW there has been the (hopefully fading) possibility that an Australian Democrat might be elected, although nobody, including the Australian Democrats, seems to know for sure what an Australian Democrat is anymore!

It is more like a lottery than a democracy.  In Tasmania's case it may be argued that the absurdities cancelled out and a reasonably democratic outcome was the product.  But if so that was more by luck than by good management, and in other states it looks like the same thing won't be the case.

19 comments:

  1. Do you know of any analysis of BTL preferencing by candidate gender?

    ReplyDelete
  2. No I don't know of any. I'd expect that gender has some influence on preferencing, in that a voter who has voted BTL for a candidate of a given gender is slightly more likely to be of that gender and slightly more likely to preference another candidate of the same gender. I have suspected a gender effect in the preferences of female candidates in local council elections under Hare-Clark - it seems to me that in such elections something like 1.5% of voters will deliberately vote for most or all female candidates, across ideological and profile lines, ahead of most male candidates.

    ReplyDelete
  3. The Australian Electoral Commission Should be publishing the below-the-line preference data as and when the data is being entered.

    Whilst this may diminish the "excitement" of the button pressing it is more important that Scrutineers and the public can be reassured that the process is open and transparent. Less technocatic and more democratic inclusive open transparency

    Tasmania is a small state and the number of voters is minimal. The results in WA Senate will be closer.

    Preliminary analysis shows that this is a State that will be determined by the allocation of below-the-line votes. Scutineers and candidates in WA have all asked for copies of teh below the line preference data files and the AEC has refused to provide it.

    Without access to this data it is impossible for scrutineers to properly scrutinise the ballot or verify the integrity of the information stored on computer.

    It is akin to shopping at a supermarket and watching the good purchased being scanned but without a receipt and a running total it would be impossible to verify if the correct amount was charged fr the items bought.


    The other aspect that is of interest is that the Western Australia results highlights a number of flaws in the way the Senate vote is counted.

    Namely. The method of calculating the surplus Transfer value, the segmentation of distribution of excluded candidates and the addition of the Droop quota (X/*y+1))

    It could be worst we could use the last bundle systen as is used in Tasmania and ACT elections

    Western Australia highligest the different outcomes as a reusl of thisese flaws.

    The system as implements was designed to facilitate a manual counting process. With the use of computer aided process these flaws in the way the vote can not longer be justified,

    We need to adopt a fully proportional mathematically correct counting system that removes the distortions and flaws that have been allowed to remain in the count taking full advantage of the use of computer based technology

    This can be best achieved by stripping back the processes of distribution and weighting the calculation of the surplus transfer value of maintain its full proportionality based on teh value of the vote not the number of ballot papers.

    The method of counting and redistribution of votes for excluded candidates should be undertaken by a reiterative process where the ballot is rest and restarted in every exclusion.

    Only surpluses should be distributed on any one iteration. Votes for excluded candidates should be redistributed as if the excluded candidates had not stood

    By resetting and restating the count on each exclusion means that full value votes will always be attributed to the first available continuing candidate. There is one transaction per candidate and no segmentation distribution where votes skipp or jump candidates at a higher value on exclusion of a candidate. the number if iterations would continue until all positions following distribution of any surpluses are filled.. Either Meek Method or the Wright system, the later being a a linear process

    The other aspect that should be reconsidered is the application of the Droop quota. why divide the cake by seven and then throw away a slice when we could and shyould divie it but six. The Droop wasted quota means that up to 14% of the electorate have no say in who should represent them

    As stated these flaws in the way the Senate vote is counted was introduced to facilitate a manual counting process and are desperately in need of review and moderisation.




    ReplyDelete
    Replies
    1. There will be a Senate reform discussion thread posted here once the results of all Senate seats are final.

      Delete
  4. Will the DoP be posted in the VTR today?

    ReplyDelete
    Replies
    1. Believe so but unsure of timing.

      Delete
    2. When the DoP comes out can you advise on Twitter when convenient please?

      Delete
    3. http://vtr.aec.gov.au/External/SenateStateDop-17496-TAS.pdf

      Delete
  5. The DoP will be fascinating to look through, and the run-off between Sex Party and Labor in the exclusion contest was so very close (the PDF is not yet available to us plebs via the AEC website).

    I wondered if you knew anything about this interesting disparity in the vote – the Queensland senate turnout. This is the numbers of House votes being reported:
    Queensland Enrolment 2,838,801; Ordinary 2,175,143; Absent 129,889; Provisional 8,527; Early Vote (Pre-Poll) 93,133; Postal 252,643; Total 2,659,335; Turnout 93.68%
    And for the Senate:
    Queensland Enrolment 2,838,801 (same); Ordinary 2,059,112 (100k lower!); Absent 72,649 (50k lower!); Provisional 11,076 (3k higher); EVPP 56,288 (30k lower); Postal 148,933 (100k lower!); Total 2,348,058 (300,000 lower!); Turnout 82.71%
    Do you know what is going on there?

    ReplyDelete
    Replies
    1. I would assume that they are still nowhere near finished counting votes for Queensland Senate, where the count has been slow compared to Tas, but that there are still uncounted ordinaries is a bit surprising. I did notice during a check of Queensland electorates last week that quite a few Senate booths still had uncounted ordinaries. Concerning provisionals, sometimes it can be ascertained that a provisional voter was entitled to vote in that state but not in the electorate in which they cast a provisional ballot. In this circumstance their Senate vote is counted but their House vote is not, and that is why the provisional figure for Senate is higher.

      Delete
    2. Thanks. It looks like things are very slow in Queensland (wind your watch back 30 years when entering the state, as the old joke goes).
      I understood the rationale behind the provisional voter count, that someone might still be eligible to vote in the state, but not a particular division – which is why I didn’t add an exclamation mark of surprise to my previous comment ;-)
      It does look like about 14 of the Queensland divisions are very much behind on Senate counting. The two lowest turnouts are currently Fairfax (54.77%) and Fisher (58.85%), so I imagine all of the effort has been concentrated on those House counts.

      Delete
  6. I think the various calculators (ABC and truthseeker) could be improved if they modelled BTLs separate from ticket votes. Or "Locked In" vs "Not Locked in". Maybe for the next election.

    ReplyDelete
    Replies
    1. Having looked at that myself (and deciding not to update my model this time around), its really quite complicated to do in practice, especially with new parties dominating the card. You end up having to make some 'courageous' assumptions in order to get a viable model, that are still more than likely to get blown out of the water when the actual results are run.

      The method that truthseeker and I used with a degree of variation exposing the possibilities is the simplest way, and isn't going to be much improved upon by throwing the kitchen sink at the problem.

      Delete
    2. Yep, I have now attempted to model BTLs separately from Ticket votes for Tas. It's on my blog - thoughts welcome and encouraged! (sure, there are some courageous assumptions, but they seem to intuitively make sense.

      Delete
  7. If Jacqui had gotten about 854 more votes (from the informal pile), she would have lost. Pretty funny really.
    http://geeklections.com.au/2013-federal-election/

    ReplyDelete
    Replies
    1. Yes I'm just writing something about that. I'm not completely convinced that it is correct, because while it would have certainly caused Sex Party to get over Labor, PUP with those extra 800+ votes might have been able to get back at the Sex Party/PUP/Liberal juncture later on, where there was a calculator gap (on existing votes) of just over 2000. But the concept is amusing enough.

      Delete
    2. Actually, having another look now. With some reasonable preference assumptions (i.e. weak flow to Sex and strong flow to PUP), there would have been too much of a gap to overcome. I'm seeing Sex losing by 1500-2000 votes to PUP (29500 to 31500 ish) after LDP exclusion. There's probably another humorous way of looking at it; e.g. If 300 ALP voters who liked PUP over SEX were tossing up whether they liked GRN or ALP first, and decided to switch to GRN instead. Makes it v/ close w.r.t. SEX & PUP. Unfortunately; not as cute to say.

      Delete
  8. This is a good result for democracy (about as representative of the vote as a magnitute-6 election can get), saving Tasmania from the "unrepresentative swill," but I really was rooting for Swan to pull it off.
    Sex didn't do quite so bad on holding the BTL votes as one might have thought. They kept 76.9% of the Sex 2 vote, plus they gained 32.5% of the Pirate 1 BTL vote (3 votes more than the Greens, and 155 more than Labor), 30.4% of HEMP (71 more than Labor), 24.7% of Shooters and Fishers (80 votes more than Labor), and as quite a surprise, even 9.4% of the BTL votes held by Madden (5 votes more than Thorp). Of course, they also got less than 10.4% of BTL votes from the Independents, No Carbon Tax, Stable Population, and Fishing and Lifestyle parties, all parties the calculator credited to them, and an abysmal 0.4% of BTL votes from the Country Alliance, who had Sex #2 above the line.
    Labor's success in holding onto the Brown, Bilyk, and Dowling BTL votes is what lost the election for the Sex Party. Antony Green's calculator projections had the leakages almost exactly right, though those flows are substantially tighter than average. I'm guessing that soft Labor voters simply didn't vote for Labor this year, leaving a higher than normal number of strongly partisan voters below the line.

    The thing that really has to hurt center-left voters in general, and the Sex Party in particular, is that the Sex Party easily could have won if the Pirates had just preferenced Swan above Whish-Wilson. It might sound like 20-20 hindsight, but the fact that Whish-Wilson wouldn't need their votes was quite obvious through the entire campaign--he was virtually guaranteed to outlast Thorp and be elected on her distribution with a quite sizeable surplus. Had they recognized this, and placed Whish-Wilson lower on the ballot, their ticket votes would have seen Swan through to Canberra. This was one of the rare instances where a tactical vote could have been seen in advance, and it could have been the game breaker.

    This may be the last federal election with party-directed preferences above the line, but if it isn't there are still several lessons to be learned.

    ReplyDelete
    Replies
    1. It would be interesting to know how many of the Madden BTLs that went to the Sex Party were actually his #1s and how many were votes he picked up at a later stage of the count. Suspect the latter had an influence on an otherwise completely illogical preference flow since FF and SXP are virtually polar opposites with SXP meant to be a kind of anti-FF in some ways. Think this data will eventually be available in the BTL data dump.

      Delete