Response From ABS to Marriage Law Postal Survey Questions

On 16 August I sent the ABS a list of fourteen questions regarding the conduct of the Marriage Law Postal Survey, in particular regarding count quality assurance issues.  Some of the questions were answered in subsequent public debate.  The response below was received today, September 11, from Michael Wilson of ABS and is reproduced in full.  My questions as sent are in italics.  My thanks to the ABS for their detailed responses at this busy time.

I have added some comments of my own below the responses, and may add more later.



ABS Responses To Questions
--------------------------------------------------------------------------------------------------------------------------

1. Will, as appears to be the case, survey responses be being processed and counted while the time window for submitting responses is still open?  
The survey responses will be data captured during the collection period, which is a common process for the ABS. The survey responses will not be finalised until after the 7th of November.

2. If yes to 1, what will happen if a respondent reports that their survey has been stolen (or that they did not receive a survey) after their vote has already been received and processed.  
If an eligible Australian reports that a form has been stolen, the ABS will mark the original barcode as invalid. The ABS will count the replacement form when it is returned, rather than the original form. The processing of valid and invalid forms will be conducted at the end of the collection period so there is no risk the survey return is already processed.

3. Will automatic character recognition be used to count survey responses?  
No. The ABS will use optical mark recognition not character recognition to identify Yes and No responses.  

4. If yes to 3, will there be manual verification (eg by checking images on a computer screen as done in the 2016 Senate election) that every survey response has been scanned correctly?
There will be processes for quality assuring and reviewing the automated mark recognition process to ensure confidence in the results.

5. To what extent will the observers appointed by Members of Parliament be allowed to observe the counting process in respect of each specific survey form?
These external observers will be nominated by two committees, representing each side of the debate. The nominations committees will comprise Members of the House of Representatives and Senators. The observers will be nominated by mid-September and the observation process will begin 3 October 2017, and continue through to 10 November 2017 (or earlier, should processing finish earlier).  Compared to the role of an election scrutineer two key differences are that observers will not have:
· the ability to view every survey form
· the ability to seek an administrative review of the survey form result validity.

6. Will every Member of Parliament, or failing that at least every party represented in the Parliament and every independent MP, be entitled to nominate their own observers?
An invitation to participate on the nomination committees was provided to each parliamentary party leader, as well as to every independent MP and Senator. Nomination of observers through two committees provides for a more coordinated and efficient system of arranging observers, than if every parliamentarian were to appoint people directly.

7. Will there be a process for observers to challenge any apparent errors they may observe, as scrutineers may do in an election, and to what level will it be possible to escalate such challenges?  
Yes. The role of an observer will be to witness randomly selected sample batches of survey responses of; yes responses, no responses and invalid responses (including those where the response is unclear). Observers will be able to specify if they agree or disagree with the survey result that the ABS has recorded. Where any doubt is raised these will be escalated to a further ABS officer.

8. Will clear and detailed validity guidelines on what will be interpreted as a valid survey response be posted online prior to the mailing of any survey forms?  
The website will have clear instructions on how to provide a valid survey response. Observers and ABS staff will have clear guidelines on the recognition of valid marks.

9. Will the survey form require the respondent to write a word or to tick a box?  
Respondents will only need to clearly mark one box.

10. If observers have the ability to observe counts being made before the window for survey responses has closed, will there be any prevention on them publishing, or causing to be published, comments about evidence they have gained in the process about the likely outcomes of the survey (without disclosing sensitive information in the process) and if not how will such behaviour (including via “leaks” to anonymous third parties) be prevented?  
Processes will be put in place to keep all information associated with the survey secret. Observers will examine a sample of the same number of 'yes' and 'no' survey responses to assure the correct recognition of marks, not the computer tabulation of totals. Observers will be required to sign deeds of confidentiality. The Australian Statistician will publish the statistical information on 15 November 2017 through the ABS website.

11. What will be the design of the envelope system used to return survey forms?  It has been mentioned that a scannable code will appear on the survey forms.  Aside from this what components (including outer envelopes) will contain identifying information, if any, whether coded or not?  
The return envelope will not have any identifying information or barcode on it. The survey form will have a scannable code.

12. Will there be a requirement for a respondent to sign any part of the returned material (such as a detachable flap on an internal envelope as used in Tasmanian local government elections) that can be used to determine whether a response was submitted by the voter or not?  
No.

13. Will there be any pre-defined provision for a request for a recount in the event that the outcome is especially close?  
No. The ABS has in place robust systems and controls for the processing, interpreting and publishing of statistical data and will be employing these systems and controls in relation to the Australian Marriage Law Postal Survey. The survey assurance process will include external observers representing the ‘yes’ and ‘no’ views. And the ABS will also engage an independent auditor to monitor a number of processes such as the dispatch of survey forms, the receipt and destruction of forms. The Australian Statistician will publish a statement on the quality and integrity of the survey on 15 November 2017.

14. It is stated that the Australian Statistician will publish a statement on the quality and integrity of the survey on 15 November 2017.  However this comes only eight days after the last date for receipt of responses.  It is possible – as is frequently the case with results by booth in federal elections - that concerns about the validity of some results might only be apparent once figures are released.  Will there be provision for investigating and officially responding to any quality concerns raised after 15 Nov?  In the event that errors in the final figures are identified, will there be provision for a revised result to be released?  
The ABS is confident it can assure the results of this statistical collection by 15 November.

--------------------------------------------------------------------------------------------------------------------------

My Comments

The response to question 2 is particularly pleasing as the previous information had suggested a very serious concern: that an unscrupulous person could illegally steal a survey form, vote and then have that stolen vote counted, disenfranchising the form's original owner if (as is likely) they did not quickly realise what had happened.  The ABS response indicates that the processes of data capture and data processing will be distinct in a way that prevents this from being a problem.  

However it is clear from the responses to questions 5, 7 and 10 that the opportunities for the observers to examine the process will be limited and will not even remotely approach the standards of normal election scrutineering.  This means that in the event of a close result, either overall or by electorate, it will be very difficult for observers to authoritatively answer any concerns that might be raised about the figures.  

Concerning scrutineering I also draw readers' attention to the concerns raised by Ian Brightwell in submission 2 to the Senate Standing Committee enquiry.  

My own concern regarding scrutiny of the MLPS is that it fundamentally differs from normal statistical processes such as the Census in view of the accuracy demands.  In a Census process, it is not important whether 18.10% or 18.11% of the residents of a particular suburb identify as Anglicans (for example).  But in the Marriage Law Postal Survey, the government's imposition of a 50% cutoff for same-sex marriage legislation to be brought on means that the normally insignificant difference between 49.999% and 50.001% could, in theory, in the end determine the issue.  

Even if the overall vote is not that close, there is also the likelihood that specific MPs may impose such a cutoff on an electorate or state basis in deciding their vote on a bill to legalise same-sex marriage.  In all likelihood the result won't be so close for the difference between a statistically accurate and an election-level accurate (and scrutinised as such) count to matter, but if it does, there will be trouble.

I may add other comments later.