(Admin note: I have just made a change to the comments screening on this site. If you have problems submitting comments but haven't had problems in the past please email me at k_bonham@tassie.net.au . Where people have trouble submitting comments I am happy to accept them by email provided it is clear which article they are to be posted to and that they are for public posting.)
I have just sent the ABS the email below regarding the upcoming (unless the High Court decides otherwise) same-sex marriage postal survey. It raises various questions about quality assurance. The email is exactly as sent except that I have removed my telephone number. (I am happy for journalists who have or can find my number to call me at any reasonable time, but I do not want phone calls from time-wasting randoms.)
Also see Michael Maley's consolidated compendium of plebisurvey issues. The entertaining issue of 16-17 year olds voting has now, alas, been knocked on the head by a fresh directive from Minister Cormann.
-----------------------------------------------------------------
Dear ABS,
I was given this email address via the phone information line (1800 572 113). Please forward this inquiry to the most appropriate people involved in the design of the Marriage Law Postal Survey.
I am an electoral analyst, political commentator and (at times) electoral design consultant who writes about elections and related processes both on my own website (kevinbonham.blogspot.com.au) and on Twitter, and who also frequently comments about electoral and political matters in the press, especially within Tasmania.
On the assumption that the High Court finds the MLPS to be legal and it proceeds, I am interested over time in receiving more detailed and preferably on the record information about quality assurance processes that will be in place, and the extent to which those might (or might not) resemble those available for federal elections and referenda.
I am well aware that things are very early in the design process for the survey and that many of my questions may not be answerable immediately. I would be happy to receive information at a later date especially if there is some prior indication of when such information might be available. I am also well aware that some of the matters I discuss could in theory be addressed by legislation or regulation, and that there may well be public statements about many of these these matters as more of the details are finalised.
I have also copied this enquiry to the Minister for Finance Senator the Hon Mathias Cormann as the now responsible Minister for the MLPS, and I am publishing this email on my website, and will report any responses I receive unless designated as off-record.
The list of questions is below.
Thank you very much for your time with what I am well aware is an unusually complex and detailed set of requests.
Yours sincerely, Dr Kevin Bonham
(phone [..])
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1. Will, as appears to be the case, survey responses be being processed and counted while the time window for submitting responses is still open?
2. If yes to 1, what will happen if a respondent reports that their survey has been stolen (or that they did not receive a survey) after their vote has already been received and processed
3. Will automatic character recognition be used to count survey responses?
4. If yes to 3, will there be manual verification (eg by checking images on a computer screen as done in the 2016 Senate election) that every survey response has been scanned correctly?
5. To what extent will the observers appointed by Members of Parliament be allowed to observe the counting process in respect of each specific survey form?
6. Will every Member of Parliament, or failing that at least every party represented in the Parliament and every independent MP, be entitled to nominate their own observers?
7. Will there be a process for observers to challenge any apparent errors they may observe, as scrutineers may do in an election, and to what level will it be possible to escalate such challenges?
8. Will clear and detailed validity guidelines on what will be interpreted as a valid survey response be posted online prior to the mailing of any survey forms?
9. Will the survey form require the respondent to write a word or to tick a box?
10. If observers have the ability to observe counts being made before the window for survey responses has closed, will there be any prevention on them publishing, or causing to be published, comments about evidence they have gained in the process about the likely outcomes of the survey (without disclosing sensitive information in the process) and if not how will such behaviour (including via “leaks” to anonymous third parties) be prevented?
11. What will be the design of the envelope system used to return survey forms? It has been mentioned that a scannable code will appear on the survey forms. Aside from this what components (including outer envelopes) will contain identifying information, if any, whether coded or not?
12. Will there be a requirement for a respondent to sign any part of the returned material (such as a detachable flap on an internal envelope as used in Tasmanian local government elections) that can be used to determine whether a response was submitted by the voter or not?
13. Will there be any pre-defined provision for a request for a recount in the event that the outcome is especially close?
14. It is stated that the Australian Statistician will publish a statement on the quality and integrity of the survey on 15 November 2017. However this comes only eight days after the last date for receipt of responses. It is possible – as is frequently the case with results by booth in federal elections - that concerns about the validity of some results might only be apparent once figures are released. Will there be provision for investigating and officially responding to any quality concerns raised after 15 Nov? In the event that errors in the final figures are identified, will there be provision for a revised result to be released?
-----------------------------------------------------------------------------------------------------------------------------------------------
(email to ABS ends)
Update (Thursday 17 Aug): While I have not personally received any answers, some of these questions have been covered by senior ABS staff in today's Senate inquiry hearing.
Answers appear (in summary) to be as follows, though I did not manage to hear the entire hearing and will be awaiting a transcript to fill them in in more detail:
3. Responses will be scanned.
9. Mark a box.
11. It sounds like just a reply-paid envelope with a form with a scannable code.
12. No requirement to sign anything.
12 is unfortunate. A signature requirement can be very useful in identifying whether multiple submission is a result of survey theft as opposed to a result of the respondent being either forgetful or fraudulent.
Of concern to me was a discussion in which the ABS representative talked about forms sent to the wrong address because an elector was incorrectly enrolled. In this case they could ask for another form and the code on the old form would be cancelled. However, when asked what would happen in this case regarding the original vote not being counted, there was the answer that it was "a question of which is returned first".
I can't think of any other interpretation than this: that if someone gets a vote for someone else and uses it before the other person realises they were incorrectly enrolled, then the misused vote will count while the voter will be disenfranchised. This might well apply to cases of vote theft too.
It seems unsatisfactory given that there are better solutions in place for postal elections, and shows the extent to which this is a rush job.
The other concerning aspect was a nonsensical passage in which the survey was claimed to be something the ABS could do in ways the AEC couldn't. The only evidence offered for this claim concerned assistance for blind or similarly impaired respondents to have their survey completed by a third party. In fact such assistance exists at AEC elections too.
Wednesday, August 16, 2017
8 comments:
The comment system is unreliable. If you cannot submit comments you can email me a comment (via email link in profile) - email must be entitled: Comment for publication, followed by the name of the article you wish to comment on. Comments are accepted in full or not at all. Comments will be published under the name the email is sent from unless an alias is clearly requested and stated. If you submit a comment which is not accepted within a few days you can also email me and I will check if it has been received.
Subscribe to:
Post Comments (Atom)
Good stuff, Kevin. Suggest you give to MP and have submitted as a question on notice.
ReplyDeleteComment from Michael Maley:
ReplyDelete---------------------------
Here are some more questions for ABS.
(1) Given that ABS has stated in the context of the survey that “There are potential penalties for providing false or misleading information to the Australian Bureau of Statistics”, will the ABS be publishing, either individually or as a consolidated compendium:
(a) all provisions of laws which in its view may be relevant to the conduct of the survey or which may impact on the rights of stakeholders or legality of behaviour of stakeholders in relation to the survey (with examples of cases to which they would apply), including relevant provisions of the Australian Bureau of Statistics Act, the Census and Statistics Act, the Commonwealth Electoral Act, the Crimes Act, the Administrative Decisions (Judicial Review) Act, any relevant State or Territory Acts, and any regulations made under those Acts and regulations;
(b) all Directions, procedures, guidelines and policy instructions to ABS or AEC staff which will structure the conduct of the survey; and
(c) all administrative forms (eg for appointment of scrutineers, or formal lodging of complaints) to be used in connection with the survey?
(2) If so, what is the planned timetable for publishing this information?
(3) Does the ABS accept that it has an obligation to conduct the survey with a procedural transparency comparable to that seen at referendums run by the AEC, and if so, what specific mechanisms to ensure such transparency will be put in place?
(4) Does the ABS accept that all stakeholders (including voters, political activists, scrutineers and the media) have a right to know precisely how each step in the survey process will proceed (as is the case with a referendum run by the AEC, where all significant steps are defined in the Referendum (Machinery Provisions) Act)? If so, will the ABS guarantee that fully detailed plans and procedures for each stage of activity will be published sufficiently far in advance of the start of each stage to enable all such stakeholders to understand the process and know what they have a right to expect?
(5) Will the ABS be publishing a Scrutineers Handbook for the survey, akin to that published by the AEC for elections and referendums? If so, what is the planned timetable for its publication?
Do I need to has a residential mailing address?
ReplyDeleteAustralia Post don't deliver to my road, AEC don't know my PO Box number, nor does ABS.
That is an interesting question. It looks like it is possible on the AEC site to update your enrolment details online to include a postal address that is not the same as your residential address. (Also, voters who live more than 20 km from a polling place can make an arrangement of this sort so that they permanently vote by post.) One would hope that where postal addresses differ from residential addresses it will be the postal address that is used. Might be worth calling AEC to check.
DeleteComment from Michael Maley:
ReplyDelete---------------------------
Re your latest addition to this post: another more than somewhat surprising statement from the ABS witnesses at the Senate hearing yesterday was the assertion that if a sample survey were to be used instead of the proposed plebisurvey process, a sample size of millions would be required.
Even if one accepts that there is a need for a stratified sample based on electoral divisions, one would expect (as a first cut estimate) that a simple random sample of say, 5000 electors per division would produce quite tight enough confidence intervals for the purpose required. Even allowing for the fact that more than 5000 per division would need to be approached to cover non-responses, it’s still hard to get the total anywhere near millions (plural). There might be some divisions in which the standard deviation of the figures was such as imply a more than negligible chance that the other option had in fact been dominant, but if that really concerned the local MP, he or she would always have the option of abstaining on a later conscience vote.
And in any case, even if 10000 people per division had to be approached, that would still on the face of it be easier to arrange, and cost a lot less, than approaching 16 million people by post.
It seems pretty clear that the ABS has adopted the universal postal model with unweighted responses because it understands that that is what the government wants. I doubt that it reflects ABS’s professional judgement about the best way of obtaining figures (and some of the questioning at the Senate hearing suggested that some Senators share that doubt). Nevertheless, the strict wording of the Treasurer’s Direction does NOT require that a postal mechanism be used, or that a large scale sample survey be eschewed, so it’s ultimately up to the ABS to defend the methodology.
While the Minister for Finance has been keen in all of this to emphasise the “independent” role of the ABS, it’s hard to have much confidence in that when they appear to have failed the first test of their independence by accepting from the politicians a rather remarkable prescription of their methodology, when quite clearly they could have pushed back.
So... that is a pretty comprehensive list of questions and I think if the ABS were called in to do the job of the AEC and either run 150 separate preferential single-winner elections from smallish geographically bound populations of around 100000 voters, or a large complex 6/12-winner partly optional preferential contest for larger populations (but with potentially 100+ candidates) then I too would demand satisfaction.
ReplyDeleteHowever... is it worth thinking of this another way?
It's clearly not an actual referendum, or a plebiscite, under the fairly strict definitions of what those entail. I think you've called it out pretty accurately as more of a survey. My question is - is that so very wrong?
The population to be sampled is the entire voting public of Australia (maybe 15 million plus at a quick guess). There are two choices - YES and NO. Under the same standards as a general election we might expect a turnout of 85-95 %, but in this case maybe we'll see a much smaller fraction of that. Possibly less than 50%? Possibly even a nightmarishly low 20%?
Would it be totally out of the question to apply the well-worn equations for margin of error and confidence interval, and conclude that a survey of 3 million people should still result in a very reliable sample (perhaps at least given some basic tests against demographic standards to identify any serious bias in responses)? Even if it were a truly regulated plebiscite conducted under the most stringent standards the government would treat the results as a guideline rather than as a binding result - not that I necessarily agree that is the right way (I would much rather a parliamentary vote for all of the good reasons) - so does it truly matter whether this is the ideal pluperfect plebiscite or an opinion poll with a staggeringly high sample size?
Some of this was covered to some degree in my first article about the postal plebiscite (http://kevinbonham.blogspot.com.au/2017/04/postal-plebiscite-australias-biggest.html).
DeleteThe concept of margin of error assumes a random sample of the whole. An opt-in sample like the postal survey isn't random, and this creates distortions that will often be a much greater source of error than random statistical noise (especially in a case like this where the "sample" is so large that the margin of error, given a truly random sample, would be less than 0.1%).
A pollster can scale their way around the problem because they know how many people of what demographic have answered a certain way, so scaling can be applied. It doesn't entirely eliminate the self-selection problem, because the way the pollster harvests respondents might still favour a certain kind of person with a certain political leaning in a way that can't be reduced to demographics. The postal survey, however, won't be retaining any demographic data that is linked to responses, only demographic data linked to turnout.
If it happens that the postal plebiscite passes every demographic representativeness turnout test that can be thrown at it, that's well and good in terms of removing any need for scaling (the possibility of hidden bias remains), but that's also extremely unlikely.
Another point here is that the government has thus far stated that a No vote will mean no parliamentary vote is held. So they're treating the result as binding (not a guideline) if the result is No by any margin, but a Yes vote by a close margin (especially if turnout is low) could open the door to all kinds of post hoc equivocation about dirty tactics or one side having an unfair advantage, the vote in particular electorates or states, the "double majority" requirement usually seen at referendums and so on.
But I betcha the "turnout" (rate of replying) won't be low. Wait and see... The people of Oz are about to tell all the lame churchy losers to stop trying to run other peoples' lives.
ReplyDelete