tag:blogger.com,1999:blog-4052593945054595675.post5701384706526531872..comments2024-03-28T14:16:10.498+11:00Comments on Dr Kevin Bonham: Oh No, This Wasn't Just An "Average Polling Error"Kevin Bonhamhttp://www.blogger.com/profile/06845545257440242894noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-4052593945054595675.post-24065755082002518492021-07-03T12:57:03.286+10:002021-07-03T12:57:03.286+10:00Thanks, I'll edit the text slightly re that.Thanks, I'll edit the text slightly re that. Kevin Bonhamhttps://www.blogger.com/profile/06845545257440242894noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-71198834413541399962021-07-03T12:46:43.091+10:002021-07-03T12:46:43.091+10:00Hi Kevin,
I thought you might find this interesti...Hi Kevin,<br /><br />I thought you might find this interesting, but I don't think the Free Range Stats model's error had much to do with incorrect assumptions on polling accuracy. Their model actually doesn't seem to use past poll accuracy as far as I can tell (in fact if I'm not mistaken, they calculate the uncertainty in their model using sqrt(2) * theoretical sampling error, i.e. sqrt2 * sqrt((p * (1-p))/n)).<br /><br />The primary reason for why their model was overconfident (as far as I can tell), is that the model assumed there was little correlated error between polls. I've been back-testing my vote model, and if I set the final 2pp prediction for 2019 to 50.3% ALP, the MoEs are, respectively:<br /><br />Assuming no correlated error:<br />ALP 48.8% - 52%<br /><br />Pretty similar to what the final FRS model said. Now, if I assume the amount of correlated error is equal to the average correlated error in polling from 1990 - 2016:<br /><br />Assuming a historically average amount of correlated error:<br />ALP 47.9% - 52.9%<br /><br />Some extra over-confidence probably also came from under-estimating or failure to model shifts in preference flows. If I turn on my preference flow shifts model (assumes that there is uncertainty in pref flows roughly equal to what they have historically been), the MoE changes to:<br /><br />Assuming historically average amount of correlated error + historically average amount of preference flow uncertainty:<br />ALP 47.5% - 53.2%<br /><br />You can probably dispute how accurate historical uncertainty is - the Greens' pref flows are probably not going to shift as much as they have in the past - but the broad idea of keeping in mind that 2pp is usually estimated from past preference flows which can and do shift (instead of treating it as if it was a sampled proportion) is probably correct.<br /><br />Hence, I think that in this case, the FRS model was probably roughly correct on the accuracy of the individual poll (although multiplying the theoretical sampling error by a factor to account for non-sampling error, instead of using historical poll accuracy is certainly an interesting approach, and I'd be interested in seeing how that plays out in future elections).<br /><br />The problem with the FRS model (and almost certainly with the Buckleys and None model too, their 2pp distribution was very overconfident and they were saved by a vote-to-seat model which assumed the map was biased against Labor: https://www.buckleysandnone.com/how-it-works-part-two/) was more of an assumption which is common in statistical theory, i.e. that repeated measurements of an unknown variable are mostly independent. In reality, when polls stuff up, they all tend to do so in the same direction, even if that direction is not predictable prior to the election, and I think it's important to account for that in election modelling.Ethanhttps://www.blogger.com/profile/07226942488996369403noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-9806422821933740012020-06-14T11:06:06.849+10:002020-06-14T11:06:06.849+10:00I agree with all that. I tend to weight the 2PP ac...I agree with all that. I tend to weight the 2PP accuracy strongly in assessing the accuracy of Australian polls because it is the figure that is most used to predict the outcome and that pollsters are judged on. But for the purposes of this article I was pointing out how the dataset used by overseas observers to say our poll failure was no big deal did not actually support that conclusion. <br /><br />Ipsos overestimating the Greens' primary instead of Labor's - something it had a monotonous habit of doing - is a good example of why major party spreads are not always ideal for judging poll errors in Australia. Another problem can be when the pollsters' reading of all the party votes is largely accurate but their preferencing assumptions are wrong, eg Queensland 2015 where most of the 2PP error was caused by preference shifting. That said the extreme preference shift seen in that election seems to be more of a risk in optional preferential voting elections than compulsory preferencing ones. Kevin Bonhamhttps://www.blogger.com/profile/06845545257440242894noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-13604932897645451032020-06-14T10:43:52.970+10:002020-06-14T10:43:52.970+10:00Hi Kevin,
Good point about the error being major p...Hi Kevin,<br />Good point about the error being major party spread rather than 2pp spread - I missed that in the article.<br /><br />Still, I wonder if major party spread is really the best way to measure polling errors in a single-member, preferential-voting electoral system like Australia's - if a pollster got the major primaries right, but completely bungled up the minor parties' relative shares of the vote (e.g. massively over/under-estimating the Greens), the 2pp could still be significantly off.<br /><br />On your note about respondent vs previous-election preferences, I think it's up to the pollster which version they use, and they should be judged on the published 2pp as well as the primary votes. After all, it is the 2pp which is reported on by the media, and it is the 2pp which makes or breaks governments. Ethanhttps://www.blogger.com/profile/06515339704982992760noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-19688207917070741922020-06-14T08:32:33.141+10:002020-06-14T08:32:33.141+10:00(note the error on 2PP spread is double the error ...(note the error on 2PP spread is double the error on 2PP - so a 50-50 poll when the 2PP is 52.74 is a 2PP spread error of 5.5). Kevin Bonhamhttps://www.blogger.com/profile/06845545257440242894noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-8435573680012909712020-06-14T08:30:45.693+10:002020-06-14T08:30:45.693+10:00ReachTEL didn't exist in 2004. Nielsen was 54...ReachTEL didn't exist in 2004. Nielsen was 54-46 to Coalition off primaries of 49-37. It's also worth noting that the database being discussed used the major party spread error, not the 2PP error, as its estimate of how wrong polls were. This is significant with Newspoll at least, which had primaries of 45-39, a major party spread error of 3.1 points, but its error on 2PP spread was 5.5 points. This was largely because Newspoll used respondent preferences that year. By previous-election preferences, the 2004 final Newspoll 2PP would have been 51-49.Kevin Bonhamhttps://www.blogger.com/profile/06845545257440242894noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-61906835712068444022020-06-14T02:20:49.943+10:002020-06-14T02:20:49.943+10:00Hi Kevin,
Interesting take, especially the point a...Hi Kevin,<br />Interesting take, especially the point about the introduction of Newspoll having significantly improved polling accuracy (till 2019).<br />I'm curious about the data you showed in this article - from recollection, the 2004 polling average was about as wrong as 2019 (due to Morgan polls showing leads for Labor when the Coalition would win by 52-48).<br />The only polls I've been able to dig up from that time are Newspoll (50-50), Morgan (51-49 and 51.5-48.5 in favour of Labor), Galaxy (52-48 in favour of the Coalition), which when averaged give a 2pp of 50.8% for Labor. Using the method you've described in the article, that's a polling error of about 7 points, which seems much larger than the 2019 error of about 5.9 points.<br />My guess is that the data set you used either contains some Coalition leaning polls, which would reduce the error, or it didn't include the Morgan polls. As I've been unable to find ReachTEL or Nielsen polling for 2004, I lean towards the former explanation; even so, it still seems like 2004 was a significant polling error approaching that of 2019. Ethanhttps://www.blogger.com/profile/06515339704982992760noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-26866887746528532862019-05-29T18:54:38.822+10:002019-05-29T18:54:38.822+10:00Kevin ,
Andrew Gelman has promised an article on i...Kevin ,<br />Andrew Gelman has promised an article on it. Can'r wait although after your efforts I can't see what he can add nowNot Trampishttps://www.blogger.com/profile/12738633092867411422noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-86410181145500701632019-05-29T14:24:53.996+10:002019-05-29T14:24:53.996+10:00Hi Kevin,
State by state polling aggregates befor...Hi Kevin,<br /><br />State by state polling aggregates before the election showed only small swings to Labor except in Queensland. This suggests a huge error in Queensland (perhaps around 15 points using Nate's calculation) and pretty small ones elsewhere. Was the problem just the one state?Tom Goldiehttps://www.blogger.com/profile/13725264333103963170noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-28692522206525998852019-05-28T21:43:31.358+10:002019-05-28T21:43:31.358+10:00Comment from Michael Quinnell:
******************...Comment from Michael Quinnell:<br /><br />*******************************************************************<br /><br /><br />Brilliant article, Kevin. I read two sites a lot - your's and fivethirtyeight. So it has been interesting to compare your in depth analysis to the so far shallow analysis of the fivethirtyeight team.<br /><br />I actually e-mailed fivethirtyeight on 14 April pleading for some preview coverage (as they regularly do, at least at a high level, of UK General elections) and mentioned that it might be useful to their US readers even to shine a light of what nationwide "instant runoff" voting looks like (what US audiences call compulsory preferential voting). However, I never heard back.<br /><br /><br />I then contacted them on evening of Thursday 23 May, after hearing a (perhaps half) joking reference to doing a live Australian podcast (I said I could arrange a venue for them). I also linked to your original polling error article, where you referenced Nate Silver's initial tweet (something I highlighted). But alas, still no response! Perhaps your new article will cause them to at a minimum re-think the haste of their initial views and look through their e-mail traffic at the same time.Kevin Bonhamhttps://www.blogger.com/profile/06845545257440242894noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-50743231334754232432019-05-28T21:43:06.808+10:002019-05-28T21:43:06.808+10:00No, hadn't seen that one yet, thanks.No, hadn't seen that one yet, thanks. Kevin Bonhamhttps://www.blogger.com/profile/06845545257440242894noreply@blogger.comtag:blogger.com,1999:blog-4052593945054595675.post-46000972694829711942019-05-28T21:31:01.130+10:002019-05-28T21:31:01.130+10:00I wonder if you've seen this article, Kevin: h...I wonder if you've seen this article, Kevin: https://www.theage.com.au/federal-election-2019/nation-s-most-influential-pollster-can-t-explain-election-disaster-20190527-p51rhc.htmlMikehttps://www.blogger.com/profile/09569372868831612962noreply@blogger.com