Well here we are again. As with the UK election, as with Brexit, as with many other voluntary voting elections we have an unexpected result with the election of Donald Trump as the next President of the USA. Pollsters are in disrepute because most had Clinton with a modest popular-vote lead, but overconfident modellers deserve their share of the blame for the level of public surprise at the result.
A few days ago, Nate Silver of fivethirtyeight was the target of a terrible Huffington Post article and an argument broke out about whether it was more accurate to say Donald Trump had about a one in three chance of becoming President, or virtually no chance at all. HuffPo was to double down with this rather pretentious piece by a stats prof accusing Silver of overstimating Trump's chances - a piece that has proved to have an exceedingly short shelf life indeed. Silver's model might not look crash hot in the wake of what has happened, but it still looks a great deal better than those that were saying Trump had only a 1% chance of winning.
Despite a few requests from readers, I didn't have time to attempt a model of the US election. To do it justice would have taken months, and I don't see that I could have done any better from afar than the more credible of those who did in fact devote such time. But I do want to add some comments about what went wrong with even the good forecast attempts (most of them anyway), as there is a lot of curiosity in my (mostly Australian) audience about this. (Australian national polls in fact got it right at another federal election this year, but we seem to be about the last place left where this still happens.)
Firstly, of course, the popular vote polls nationwide for POTUS were wrong, but not by a very great amount. Final polls and aggregators had the margin for Clinton over Trump at around 3.5 points. In the current primary count Clinton has a tiny popular-vote margin which is currently at around 0.2 points and which may finish up around 0.6 points.
(Note added: Some sources are now projecting Clinton's popular vote margin as over 1 point and perhaps even closer to 2, which if true further underlines the point that national polling error was not that large by historic standards.)
FiveThirtyEight argued repeatedly that this was quite a probable scenario, having Clinton winning the popular vote but losing the Electoral College at a 10.5% chance in their final model, compared to just 0.5% for the reverse. The basis for this was that the Electoral College map was slightly unfavourable for Clinton.
Why the national polls went wrong will be the subject of much enquiry in the months to come. People have put to me the idea that it was "shy Trump voters" that did it, and there is lots of ammunition in the difference between live phone polling and other methods for that conclusion. But it could well also be that polls overestimated turnout for Clinton and didn't realise how many left-wing voters, still deflated by Clinton's use of the party machine to just overcome Bernie Sanders, might have said they were voting Clinton but then simply not shown up.
FiveThirtyEight noted that a national three-point polling error in Trump's favour (about what happened) should lead to a more or less tossup race. Had there been a uniform swing in the popular vote margin of the size that we have seen, Clinton would have lost Nevada but saved Wisconson, Michigan, Pennsylvania and Maine-2, and scraped home with a 273-265 Electoral College margin.
Swings are never uniform, so I tried simulating a normally distributed 2.9 point polling error with a randomised swing with the same standard deviation by state as the one that actually happened. (In calculating the SD for this purpose I ignored one massive outlier, Utah). When I ran Monte Carlo simulations of this (10,000 runs) Clinton still won the election 63.3% of the time, and only lost the EC as badly as she seems to have done (232-306) or worse 2.6% of the time. With a more "fat-tailed" distribution I would have got slightly different results but still the picture that this was not a likely result given the national popular vote and without something unusual happening would remain.
The discrepancy isn't explained by swing states being swingier (they weren't). The best description for the cause of Clinton's loss is that she was undone by the regional unevenness of the swing that affected the three key states described above. These were seats Clinton was sometimes viewed as taking for granted while Trump's tilting at them appeared hopeless. However in strategic terms Trump had no reason on earth to care what margin he might lose by, while for Clinton it was important to not just win but win big so that her success would carry through to down-ticket races. With polling showing Clinton winning Wisconsin especially by several points it may be that the Clinton team thought key swing states were in the bag.
Apropos of nothing much but hopefully interesting, this is what FiveThirtyEight projected about how the 2012 results would relate to shifts in the popular vote margin in particular states:
and this suggests the polling did show it) - the model expected swings to Trump in seats won by Obama and swings to Clinton in seats won by Romney. Now for what actually happened:
This is either a sign that his campaigning in the key states was better than Clinton's, or that the key states were always going to deliver for him. I lean more towards the former. There is evidence that especially in Wisconsin, which had the biggest polling failure of these states, late deciders went very heavily for Trump - though this does not alone explain the polling failure.
There's a question here about the extent to which bad modelling played a role in convincing left-wing voters that the outcome was a done deal, fuelling complacency and hubris among Clinton supporters. Nate Silver was right when he said Grim's article was "irresponsible". I am not convinced that there was actually a way for objective polling-based modellers to get this right given the problems with the polling in swing states. But giving Trump a 1% or even 10% chance instead of at least a 20% chance was completely inexcusable stuff from those who did it.
"Senator, you are no Donald Trump"
Lastly I would like to comment on some remarks by Tasmanian Liberal Senator Eric Abetz, in the knowledge that by his own standards he and anyone who takes them seriously will ignore me anyway. Speaking on radio this morning, Abetz claimed that the result showed that the public respond to conservative values and that right-wing politicians should ignore commentators and continue banging on against same-sex marriage, section 18C of the Racial Discrimination Act and so on.
He might have a point had the US elected a President Huckabee or Santorum. Instead, Abetz's brand of religious political culture-warring was almost nowhere to be seen in the final campaign, its adherents having been sent packing with no support at a very early stage of the Republican primaries. While Donald Trump did throw red meat to the base on issues like SCOTUS appointments and abortion, and while he did pick a religious conservative as a running mate, same-sex marriage simply wasn't an election issue.
For the most part Trump campaigned less like an Abetz-style Christian conservative and more like an avatar for most if not all of the seven so-called deadly sins. Genuine "Christian conservatives" were initially much taken aback by this, many saying they couldn't possibly support him, but they tended to come back to the partisan fold over time and argue that however crass Trump was, Clinton was worse.
The result did show that US voters were prepared - at least when the alternative was yet another dodgy political-class dynasty - to embrace a candidate who said what he would, no matter who he offended. That's a different thing to embracing candidates who continually bang on about the right to be offensive instead of focusing on issues voters actually care about. It might be argued that the Left has paid a price for too much culture-warring as an alternative to policy focus in this case, but the Right is far from immune from the same.