Thursday, September 26, 2019

Wonk Central: The Hare-Clark Recount Bug and the Wangaratta Case

Welcome back to Wonk Central, this site's sporadic series of articles that have been deemed too mathsy, too quirky or too niche for remotely normal human consumption.  In this case, it's clearly all three.

In this episode we take a very close look at the Hare-Clark Recount Bug (which could also be called the Hare-Clark Countback Bug, but "recount" is the term confusingly used for countbacks in Tasmanian law). What is it, why don't we kill it, and is the minister aware of any alternative approaches?  The impetus for this article is a recent court case in Victoria, in which a candidate disadvantaged by the bug in a Wangaratta Council countback in 2017 took legal action but lost.  Among other tries, the plaintiff (a local doctor, former soldier and Australian Country Party candidate in last year's state election) claimed that the use of a countback method that disadvantaged him deprived him of the human right to take part in public life.  

For various reasons, the judgement didn't get into the weeds of whether the countback system in use was fair or whether there was any better alternative.  Therefore, let's go there here.

The system and its origin

The principle behind recounts in Hare-Clark and other similar multi-member systems is that when there is a casual vacancy, the new member who fills that vacancy should, to the greatest extent possible, be a like-for-like replacement for the member who has resigned or died.  (See a good general explanation on the PRSA (Vic-Tas) page.) The argument for this is that this best preserves the intention of the original voters.  

The method by which this is done is - in simple terms - to redistribute the votes that the departing member had at the point of their election among the candidates contesting the countback.  (In practice it is a little more complex than that, because of adjustments to bring the departing member's votes to quota or as close to it as possible, but the idea that it is only the departing member's votes is the most important point.)

This system of countbacks, with minor changes along the way, goes back 100 years.  In the very early days of Hare-Clark in Tasmania, casual vacancies were filled by single-seat by-elections.  However these frequently changed the composition of the House, a bigger issue in finely-balanced proportional-representation parliaments than in single seat systems where large seat margins are common.  (A further reason not to use single-seat by-elections, that they would tend to replace minor party MPs with major party MPs, had yet to become as apparent as it is today).   

The countback system, "given to the Committee in evidence by a certain Mr Samuel Bind, Coach Builder of Burnie", has been hugely successful at delivering like-for-like replacement at party level in Tasmanian state politics.  In 100 years of it, there have been just three cases where its use has resulted in a change of party composition:

* Denison 1983, a well-known case in which Democrat Norm Sanders' seat was taken by independent Bob Brown.  Both had been high-profile anti-dam campaigners and Sanders' voters had tended to preference Brown higher than Sanders' very low-profile Democrat running mates.

* Braddon 1961, a much less well-known case in which ex-ALP independent Reg Turnbull resigned and his countback went to a Labor candidate rather than his independent running mate.

* Clark 2019 (recount for Denison 2018), in which former Labor MP Madeleine Ogilvie, who had contested the election as a Labor candidate, won Labor MP Scott Bacon's countback but then chose to sit as an independent. (See detailed coverage.)

The first two cases are simply examples of the system giving voice to a voter view of like-for-like that happened to cut across party (or independent group) lines.  The third highlights a possible weakness of the system - using defeated candidates from a previous election to fill vacancies may mean that vacancies are filled by candidates disenchanted by losing to within-party challengers.  However it can't be too common an issue, since this is the first time such an MP has defected.

The system, clearly successful based on its major objectives in Tasmania, has been exported to local councils.  But local councils often don't have strong party systems, and the system seems to have been more criticised in council contexts, especially outside Tasmania.

So where's the problem?

The problem canvassed in Dr Fidge's unsuccessful Wangaratta appeal involves the way in which countbacks can favour some candidates who failed to be elected in the original election over others.  The cases in question involve a countback for a vacating member (who may have resigned or died) where the vacating member was elected at a later stage of the distribution of preferences.

In this Wangaratta case the quota in the 2016 election had been 1971 votes in a race for 4 seats.  The primary votes were

Ken Clarke 1950
Dean Rees 1524
Julian Fidge 1271
Dave Fuller 1004
Ruth Amery 920
Ashlee Fitzpatrick 876
Greg Mirabella 835 (there's a familiar name!)
George Dimopoulos 619
Russell Stone 447
Luke Davies 406

Nobody had quota so Davies was excluded.  This put Clarke over quota with a very small surplus, which was followed by the exclusions of Stone, Dimopoulos and Mirabella.  Mirabella's exclusion gave Rees a small surplus.  After all this the remaining candidates were:

Amery 1603
Fidge 1553
Fuller 1448
Fitzpatrick 1296

Fidge had already fallen to fourth.  Fitzpatrick's #1 votes now took Amery to 2061 (well over quota) and Fitzpatrick's remaining votes put Fuller over quota as well, with Fidge left in the count unelected in fifth place with a final tally of 1734. We can note from this that Fidge attracted a strong primary vote but, in this field, few preferences.  

The 2017 countback

Councillor Amery died suddenly in 2017 and her place was filled by countback of her 1971 votes (this total is reached by slightly reducing the value of her votes received from Fitzpatrick to bring Amery down to quota, although they were not reduced in value in the original count.)   

Amery's 1971 votes included her 920 primaries, which in theory could have flowed next to any of the six countback candidates.  But they also included 1051 votes that had been #1 votes for Davies, Stone, Dimopoulos, Mirabella or Fitzpatrick - but not Fidge because he was never excluded.  On the countback, all these 1051 votes returned, at least initially, to the candidate they had been #1 votes for.  

After throwing all Amery's votes to the candidates, votes stood at:

Fitzpatrick 652
Mirabella 445
Dimopoulos 323
Stone 285
Davies 165
Fidge 95

Fidge was first excluded from the countback and Fitzpatrick went on to win.

Even had the other candidates not got any of their first preferences back (ie had only Amery's first preferences been used in the countback), Fidge would not have won.  Only 10.3% of Amery's primary voters had preferenced him above the other five countback contenders and Fidge would have been eliminated in last place or second-last anyway.  Also, had the countback been just of Amery's primary votes, Fitzpatrick would still have been the initial leader, and would probably have won the countback.  And even had Fidge polled a better share of Amery's primaries, his performance on preferences in the original election suggests he still would have struggled.  So in this case there is not a real question of the countback having failed to pick the most like-for-like replacement.  

It is, however, a great illustration of how it could, with Fidge's opponents getting an average of 210 votes back at the start of the countback that Fidge had no immediate access to.  If the idea is to find the best like-for-like replacement for Amery, why is a vote that is, say, 1 Stone 2 Amery fully relevant to that consideration, while a vote that is 1 Fidge 2 Amery is irrelevant?  Sure, the former was among the votes that elected Amery while the latter was not. However, worse than this just being an accident, Fidge is being punished in the countback for getting too many votes in the original election! 

The other thing worth noting here is that Fitzpatrick (who won the recount easily) was also slightly disadvantaged by the bug.  In the original election she gave 458 of her primaries to Amery, but in the countback she only got 368 back.  

Why not redo the whole count without the vacating member?

A 2014 review of the Victorian submission picked up a range of other discontents about using the original election to conduct recounts.  Especially, countbacks can often elect obscure candidates who polled poorly in the original election, especially where a councillor is elected on a vacancy then quits, triggering a second countback.  The Municipal Association of Victoria submitted:

"Currently, if a councillor resigns, the only votes that are counted in a countback are the votes forming part of the resigning councillor’s quota. Presumably this was designed before the advent of computers to reduce the labour intensiveness of conducting a recount.

The drawback of the current approach is that it does not deliver the most democratic result or reflect the voting intention of all of the electors who cast a vote in the particular contest.

For example, if Councillor X was elected over candidate Y by a small margin (eg. 1 vote) for the last
position to be elected in a particular ward at the general election, in a subsequent recount, the only
votes which would be counted are those votes which comprised Councillor X’s quota. This would almost certainly lead to an outcome where some other less-preferred candidate to candidate Y was elected, despite candidate Y sitting on 99.9% of a quota. I.e. even if candidate Y won 49% of Councillor X’s quota compared to candidate Z winning 51%, candidate X would be elected despite candidate’s Y original votes (one vote short of a quota) remaining untouched and uncounted on the table. This produces an anomaly where votes for candidate Y are disenfranchised and votes for Councillor X are given a disproportionate and unnecessary influence."

As the link to the Tasmanian history above shows, the MAV's submission is wrong about pre-computer labour intensiveness as a factor.  Rather, the original aim was like-for-like replacement as a replacement for by-elections with the express aim of preserving the stability of party numbers, and there is no evidence that a countback of all the votes was ever canvassed.  

The MAV's submission is also overdoing it in saying that the candidate disadvantaged by the bug will almost certainly lose.  In a case like Wangaratta, the candidate starts without direct access to about half the votes.  But if they poll well enough on the remaining votes to survive the initial cut, they can then start picking up preferences from the #1 votes of excluded candidates in the countback.  A candidate who is strongly preferred by primary supporters of the vacating candidate can still win.  

Indeed, when I went looking through Tasmanian recounts I hadn't looked at before for an example of this, the very first one I checked was pretty relevant.  In Break O'Day at the 2018 council elections, John Tucker (now a state Liberal MP) was elected late in the piece, and Margaret Osborne was the last unelected candidate left in the count, missing out by less than a tenth of a quota.  Osborne was fully disadvantaged by the recount bug, and five of her opponents were not disadvantaged by it with a sixth partly disadvantaged.  However Osborne won the countback very easily.  An important difference was that a smaller share of the available votes in that case were affected by the bug.  [Update 2020: Jess Greene's win in West Tamar is another example, with nearly half the available votes affected,]

It's also a little misleading to refer to candidate Y's original untouched votes.  The reason is that not all of those votes started with candidate Y.  In a full countback, the order of exclusions might change and some of those votes might never reach Y in the first place.  And it's very misleading to call candidate Y's voters disenfranchised.  One is not disenfranchised just because one's vote landed with a candidate who didn't win in the original election or because one gets no say in who replaces them; it just means elections have winners and losers.  

But in any case, this opposition to vacating-candidate countbacks gained currency and the Victorian Government was intending as of 2018 to replace them with full recounts, a course opposed in submissions by the PRSA (Vic-Tas) and Victorian Electoral Commission.  I'm not exactly sure where the proposal stands now but the issue may have been overtaken by the Victorian Government's current, and even worse, idea of reverting to single-member wards for most councils.   

Aside from labour issues (irrelevant if ballots are data-entered) there are two big problems with redoing the original count with the departing member's votes redistributed:

1. It violates the principle of like replacing like

In a contest where a candidate has narrowly beaten a politically opposing candidate for the final seat, a full countback for that candidate's seats will often elect the opponent.  While it can be argued that this is the revised intention of the voters, the unwelcome aspect of it is that it may deter an incumbent from resigning in a case (such as a scandal or ill health) where resigning is a good idea.  The principle of like replacing like usually encourages incumbents who should resign to do so, knowing that someone similar should replace them and the balance of their council or parliament should not be affected. 

2. It can "un-elect" original winners

In a full countback it can sometimes happen that a candidate who won in the original election, and wants to continue, ceases to be a winner.  This could have happened to Tasmanian Senator Nick McKim, and who knows what the High Court would have done about it.  It did happen to Melbourne City Councillor Michael Caiafa, albeit with Group Ticket Voting nonsense a factor in that instance.  

A good Hare-Clark example is Denison 2010.  Elise Archer (Lib) outlasted Richard Lowrie (Lib) by 61 votes.  Archer then went on to outlast Andrew Wilkie (Ind) by 315 after Greens preferences did not flow quite strongly enough to Wilkie.  In a recount minus some originally elected candidate other than Archer, it's possible Lowrie would have edged out Archer at the key point because of changes to the values of votes, thereby "unelecting" Archer.  It's also possible that having done this he would have eventually lost to Wilkie, so (for instance) a Labor vacancy could cause a Liberal seat to fall to an independent.

What would have happened in a full countback anyway?

The court finding presents as factual background that Fidge not only got more primary votes than Fitzpatrick, but also had more votes after the distribution of preferences (1734-1296).  But the latter is a useless statistic, because Fidge's total of 1724 includes votes he received from Fitzpatrick after her exclusion (and also, since Fitzpatrick was excluded, her final score in the preference distribution was actually zero).  We know that in the original count Fidge was 258 up on Fitzpatrick when she was excluded, but that does not mean he would beat her in a full countback with extra votes added to the count.

The relevant question regarding a full countback is the point at the original count where Davies, Stone, Dimopoulos and Mirabella had been excluded and Amery, Fuller, Fitzpatrick and Fidge were competing for two places.  In a full countback, the 1599 votes that originally went to Amery would have been distributed to other candidates and ultimately settled with one of Fuller, Fitzpatrick and Fidge.  At this point Fidge was 106 ahead of Fuller and 258 ahead of Fitzpatrick.  We know that Fidge was a much weaker performer than Fitzpatrick on Amery's primaries, and also that Fidge performed very weakly on Fitzpatrick's primaries compared to both Amery and Fuller, getting only 10.8% of them in a three-way throw.  If Amery's preferences flowed anything like Fitzpatrick's, then the most likely result of a full countback would be both Fitzpatrick and Fuller jumping Fidge and Fidge finishing fifth again.  

Another possibility in theory would be Fitzpatrick jumping both Fidge and Fuller without Fuller passing Fidge, in which case Fuller would be unelected.

Other solutions

Besides full countbacks, which I think should be rejected for the reasons stated above, here are some other possible options for filling Hare-Clark and similar casual vacancies:

Party nomination

This is the procedure used (subject to what is usually a state parliament rubber stamp) for the Senate.  It is appropriate for the Senate because the regimented ticket order with above-the-line votes flowing down the ticket means that some positions on any ticket are likely to be uncompetitive and parties are likely to nominate ticket-fillers rather than serious candidates for those positions.  Another arguable advantage for federal politics is that parties can run star candidates for risky Lower House seats and still have the option of using them to fill Senate vacancies if they fail.  However, for Hare-Clark the point against it is that the party is appointing a candidate with no electoral mandate, while the original election provides a way of picking a candidate with a mandate from the election's voters.  (That said, in Tasmania parties sometimes deliberately run filler candidates to concentrate their vote in their incumbent MPs, and then struggle if those fillers are later elected on countback.)

Mixing in the leftover quota

To avoid the issue of unelection it's sometimes argued that the votes from the leftover quota (in this case the votes Fidge finished with and a few from surpluses of Fuller and Amery) should be included in the countback as well as the vacating member's final votes.  However, this violates like-for-like replacement, probably much more seriously than full countbacks do.

Just use the vacating member's primaries (and perhaps the surplus votes they receive from candidates who make quota before anybody is excluded)

This solution would get rid of the countback bug and put all countback contestants on an equal footing in the race to be deemed the most like-for-like replacement for the vacating member.  But it also disadvantages those voters who voted 1 for a candidate who was excluded early and preferenced the vacating member, since their vote no longer has any impact on the replacement of that member.  In some cases, that can be most of the vacating member's voters.  It increases the problem of countback candidates winning with a very small mandate.  

Include votes for failed candidates that would have flowed to the vacating member had those candidates been excluded before the vacating member was elected

This seems like an obvious try but it is fraught with messy problems.  One is that for a vacating member who was elected early in the exclusion process, the countback may end up swamped with votes for unsuccessful candidates and become much less representative of the near-quota of voters who voted for the vacating candidate.  Another is that many votes of this type will have gone on to elect somebody else so those voters would get two bites of the cherry.  A third is that whether or not a vote would have flowed is obvious enough if it is a second preference for the vacating member but can be difficult to identify otherwise.  

(Oddly, what's suggested here does happen to a limited degree in cases where the last candidate is elected without quota because of exhaust.  The final losing candidate has their votes thrown to try to bring the vacating member up to quota, a peculiar procedure which can help reduce the impact of the bug on that losing candidate.  To give an idea of just how weird this piece of quota-fetishism is, it can result in votes being included in a departing member's recount when the voter put the departing member last.)

It's not a bug, it's a feature!

Lastly there's the suggestion that there is actually nothing much wrong with the recount procedure as it stands and it actually should be as it is.  I've seen people say this pretty often, though I don't remember the arguments used or who exactly made them.   Anyone of this mind is welcome to have a go in comments.  

I can't promise a Fields Medal to anyone who provides a satisfactory improvement to the recount procedures, but the number of Hare-Clark junkies who would consider the solver of this problem Very Clever could well number in the dozens!  A few electoral commissions would be interested too ...


  1. I think the goal should be minimizing wasted votes. That would mean taking all the votes that didn't elect someone (or in the case of votes for the departing candidate, no longer elect someone) and counting back with those. I believe this is the "Mixing in the leftover quota" solution above. I think whether or not the candidate elected is a "political opponent" is for the voters to decide: Fidge would only win that count if enough voters that elected Avery saw him as politically aligned with her.

    1. The reason I don't support anything that is likely to elect a diametrical opponent as perceived by the outgoing member is that it can deter a member from resigning when they should resign. In the Victorian system the candidate left last unelected will indeed need some preferences from the departing member's quota to win with the leftover quota mixed in, but the number they need may be trivially small. In other versions of the system like Tasmania's where votes can exhaust (this seems not to be the case in the Vic system), they may not need any votes from the departing member's quota at all.

  2. Is it as simple as conducting the count as normal...
    1. The vacant candidate's votes are distributed at full value as Distribution #1
    2. The count proceeds as normal electing the full suite of candidates
    3. Previously elected candidates cannot be eliminated nor have another candidate elected at their expense

    Say 5 candidates with 3 to be elected and quota 40.
    The primaries are as follows:
    A 50 (100% prefs: A-E-D)
    B 40
    C 33
    D 27
    E 10 (100% prefs E-C)
    So A & B are elected off initial quota, and C elected off E's preferences.

    BUT if A resigns, then do a recount using my 3 steps:
    Count 1
    A 0
    B 40 elected
    C 33
    D 27
    E 10+50=60 elected
    So E & B elected off initial quota.

    This leaves
    A 0
    B 40 elected
    C 33 + (10/60×(60-40)) = 36
    D 27 + (50/60×(60-40)) = 44
    E 40 elected
    Doing a "normal" election here would elect D, hence unelecting C (like Caiafa in Melbourne)

    But applying my rule 3 above, you would instead re-elect C. So B & C retain their seats and E gets A's seat. This is fair as it avoids inadvertent or strategically advertant unelection. It reflects the will of voters as A's preferences went to E so it shouldn't change political compositions.

    1. This would remove the unelection problem but the composition-change problem would still remain in cases where the final seat was won by a very narrow margin and the vacancy is for the party that won the final seat. For example, if the 2016 Senate election for Tasmania was rerun by this method for a vacancy involving either Green, I would expect One Nation to take the final seat reducing the Greens to one (the Greens, with two high-profile candidates and one who at the time was low-profile, beat One Nation by 141 votes, and not every voter preferencing the high-profile Greens candidates also preferenced the third Green). Another example could be the Tasmanian state seat of Franklin 2018 - the incumbent Green Rosalie Woodruff retained her seat by a narrow margin. If she resigned then the #2 candidate might not do well enough to win the full countback in her stead, because some 1 Woodruff votes would have leaked to other parties.

      I am interested in the idea of unelection-proofing for full countbacks though because this could be useful for dealing better with disqualification cases (including the Caiafa case). In a disqualification case it doesn't matter if political compositions change at the expense of the disqualified MP, so long as it isn't at the expense of anyone else. So I assume the method to achieve this would be what you suggest and would be as simple as follows: if an originally elected MP is running last in the count, then they are not excluded, and instead the candidate running second-last is excluded. Also, as soon as any candidate originally not elected makes quota, this determines the result of the countback since the other places will be filled by protected candidates. So in your example with E getting a quota on primaries in the countback, the countback can be stopped immediately. If more than one originally unelected candidate made quota on primaries (or at the same count later in the countback) then presumably the one with the most primaries would be elected to the vacancy and the countback would stop immediately.

  3. Is there rules in the Federal Senate as to what happens to the seat of an unaligned independent Senator?
    It seems that though doing the recount will pretty much always elect someone from the opposition, which is why to me anyway some sort of group ticket seems a neat sidestep!

    1. If an independent Senator elected as such resigns or dies (thus creating a casual vacancy) the state parliament can nominate whoever it likes to fill the vacancy. Probably after the events of 1975 there would be outrage if that person was too different to the original Senator.

      Recently Cory Bernardi resigned while being an independent, but he had originally been elected as a Liberal so the replacement must be a Liberal.

      If an independent elected as such is disqualified such that their original election is annulled, it would be a special count as with all the other disqualified Senators. To have an above the line box the independent has to run in a group with at least one running mate, who might or might not win the seat instead depending on how close the margin was and how strongly the ticket held or didn't hold.

  4. An overall flaw with the HareClarke system is that everyone has only 1 vote independent on whether 1 position is to be elected in case of mayor or deputy mayor or 12 councillor positions.
    To make the voting system fair every voter would need to have the number of votes corresponding to the number of positions. This would also eliminate the recount bug problem as the only change would be the votes for the candidate resigning while all other votes would remain counted.

    Currently for a vote to be valid one has to label the ballot paper 1 to x with x being the number of positions available. This does create the illusion that we are voting in the case of the city council election for 12 people when in fact we have only 1 vote for 1 position out of 12 in this case.

    I didnt follow this last year however the george town council had a by election rather than a recount. By elections are more expensive i guess and also would lead to some of the elected members may not retaining their position.

    In any case i do believe the best representation of who voters would like to see as the member of the council or parliament would be to give each voter 1 vote for each position available and the top candidates with the most votes will gain office. If one candidate resigns the candidate with the next highest votes attains office if they are still interested and able to take up office.

    1. George Town had a by-election for one position because its mayor resigned from council. When this happens they have to have a by-election for mayor so they take the opportunity to have one for councillor as well.

      Giving voters 12 votes for 12 councillor positions might be similar to the Senate system in the first part of the 20th century, or variants including preferences might be attempted. In either case though it tends to result in a majority faction being able to win all the positions and minorities not getting represented.

  5. What if, after the original election count is finished, you do a special redistribution of the last-person-standing's votes to all the elected candidates. Then when there is a recount, a portion of their votes will be in play along with everyone else?

    1. That is an interesting suggestion and I'm going to think about it carefully over some time, as it at least seems to help the final disadvantaged candidate to some degree without obviously over-helping them. It can also help other disadvantaged candidates to a smaller degree, eg a vote that is 1 for the second last loser, 2 for the last loser and 3 for the vacating candidate goes back to the second last loser.

      Something vaguely like this is done in the existing system in the special case where the vacating candidate was elected without quota - prior to the recount starting, the last candidate is excluded and their preferences distributed between all candidates elected without quota in order to try to bring the vacating candidate up to quota. This can have the very bizarre impact (which your suggestion avoids) that a vote that had the vacating candidate absolutely last can sneak into the vacating candidate's recount.