"Margin Of Error" Polling Myths

A lot is said about "margin of error" in public discussion of opinion polls, and nearly all of it is wrong.

This is another piece I've been meaning to write for quite a while.  There are many other articles about this on the web, but I'm not aware of any that make all the points I'd like to make and make them in an Australian context.

The concept of "margin of error" is one that is commonly talked about in discussion of polls.  It is often used to (rightly) deflate breathless media babble about movements in polls that are supposedly caused by whatever the media pundit wants to see as the cause, but in practice are often nothing more than random variation from sample to sample.  Possum's Trends, The Horserace and Random Numbers (2012) was a classic debunking of that sort of commentariat nonsense, written in the days of the older, much bouncier Newspoll.

Unfortunately, of all the things that get talked about in polling, margin of error is probably the concept that best shows that "a little learning is a dangerous thing".  People grasp the basic principle and misapply it constantly - and are assisted by pollsters, the media and even the Australian Press Council's otherwise good (and too often disregarded) poll-reporting guidelines in doing so.

Why Is Seat Polling So Inaccurate?

The accuracy of Australian seat polling has been an important topic lately, especially given the coming by-elections.  By-elections are very difficult to forecast.  Even after throwing whatever other data you like at them (national polling, government/opposition in power, personal vote effects, state party of government) they are less predictable than the same seats would be at a normal election.  So it would be nice if seat polling would tell us what is going to happen in them.

Unfortunately single-seat polling is very inaccurate.  I discussed this in a recent piece called Is Seat Polling Utterly Useless?, where I showed that at the 2016 federal election, seat polling was a worse predictor of 2PP outcomes than even a naive model based on national polling and assumed uniform swing.  The excellent article by Jackman and Mansillo showed that seat polling for primary votes was so bad that it was as if the polls had one sixth of their actual sample size.  It doesn't seem that seat polls are useless predictively, but we certainly can't weight them very highly.