The aggregate is essentially a weighted average of two-party preferred polling derived from all recent polls that are considered of sufficient standard. 2PP figures from each poll (see "The 2PP score that is used" below) are multiplied by various weightings based on their recency, accuracy and other issues, and then the sum of the multiplied poll scores is divided by the sum of the weightings.
Pollsters Included:
Polls by the following pollsters are included when available: Galaxy, Newspoll, Nielsen, ReachTEL, Morgan, Essential, AMR, Ipsos. Other pollsters may be added subject to their completion of three polls of national 2PP voting intention by the same methods, and adequate methods documentation. Even if a poll is by an included pollster, the poll is not included if it is an "internal poll" (defined as a poll commissioned by a political party, union, business or lobby group).
The 2PP score that is used:
For ReachTEL polls (which release primaries to one decimal place but 2PPs rounded to the nearest whole number) the 2PP used is found using the formula:
Coalition 2PP = Coalition Primary + .17*Green Primary + .537*PUP Primary + .53*Others Primary - 0.14
(This is derived from last-election preferences. The -0.14 is an adjustment for the impact of three-cornered Liberal/National contests).
This is rounded to one decimal place, with the constraint that if the number thus found doesn't round to the released 2PP, then the figure is altered to the closest value that will round to the released 2PP.
For other polls the 2PP that is used is the average of:
1. The last-election 2PP released by the pollster
2. The calculated 2PP based on the released primaries (formula above). (If PUP and Others are not separated but lumped as Others, then the PUP and Others terms are replaced by .533*Others).
If the pollster does not release a last-election 2PP (eg May 2016 ReachTEL) then just item 2 is used.
This is rounded to one decimal place, with the constraint that the maximum allowed difference between the number thus found and the pollster's released 2PP (prior to applying any house effect correction) is 0.2 for Morgan, and 0.4 for all remaining pollsters. So if a Newspoll released 2PP is 51, the aggregated 2PP will be in the range 50.6-51.4.
For polls in late 2013 that were released with 2PPs based on 2010 election preferences, a 2PP was derived from the primaries using the method for ReachTEL stated above, save that the released 2PP was completely ignored.
Time weighting:
The aggregate is designed to be fairly cautious in adjusting to apparent changes, in case these are actually caused by individual aberrant results. The base time weighting is 5 for a poll released this week, 3 for last week, 2 for the week before last, 1 for the week before that. At present a new polling week is considered to start from Saturday with the reset being applied from midnight Friday. Intermediate weightings of 4 for a poll bridging this and last week, 2.5 for a poll bridging one and two weeks ago, 1.5 for 2-3 weeks ago, 0.5 for 3-4 weeks ago, are used in the case of polls that use rolling data for two weeks (a la Essential and usually Morgan) or that derive their data entirely from the middle or end of the previous week (eg ReachTEL often polls on Thursday.)
So for instance in the typical week the new Essential carries a weighting of 4 when it is released, but this is reset to 2.5 when the next polling week starts. A ReachTEL released on Thursday or Friday based on polling on Wednesday or Thursday is weighted at 5 but this is reset to 4 when the next polling week starts.
During election campaigns, in the last three weeks a weighting of 8 is applied for polls based entirely on data sampled (not released) in the last four days. In the last week a weighting of 10 is applied for polls based entirely on data sampled (not released) in the last two days. Only the most recent poll by any pollster can qualify for these weightings.
Number of polls included:
Only the two most recent polls by any pollster are included. In the case of Essential, only the most recent poll is included.
Quality weighting:
Pollsters are weighted based on my view of their performance. A poll has a start weighting of 1 and bonuses, penalties and ceilings can be applied:
+0.1 Pollster judged best pollster of the previous election
+0.1 Long established pollster successfully using same stable method at multiple elections
0.8 Ceiling: Applied if pollster has produced less than eight national polls by any method
- penalty (set level): Applied for poor performance or tracking or irregular behaviour of whatever kind over a prolonged period. If exceptionally severe a poll can be excluded altogether.
The current quality weightings are:
1.2 Galaxy (live phone only)
1.15 Galaxy (live phone/internet)
1.1 Newspoll (includes both old Newspoll and Galaxy-Newspoll), Nielsen, Morgan Phone (but see below)
1 ReachTEL, Ipsos
0.8 AMR
0.7 Essential, all fortnightly Morgans that include face-to-face and/or SMS
0.6 All weekly Morgans that include face-to-face and/or SMS
If only the 2PP is released for a poll and not the primaries (a common practice with Morgan Phone) then the quality weighting is halved.
Note that Essential is also effectively penalised by having only one poll included at any one time. Unfortunately its 2013 election tracking performance was rather bad, and especially so when considering that its rolling-average method is designed to smooth out week-to-week bouncing, so in theory Essential should provide better tracking than other pollsters.
For penalty to Morgan see list of changes below.
For my full review of pollster performance at the 2013 election see here.
Respondent vs Last-Election Preferences:
Morgan publishes polls with respondent-allocated preferences as the headline figure. This actually worked quite well at the 2013 election in Morgan's final polling, because voters for the Greens and "others" were slightly more ALP-friendly than in 2010. Normally, however, it is both a less accurate way of allocating preferences and a more volatile one. Therefore, any respondent-allocated preferences that are published are ignored. The last-election preference 2PP is considered to be the pollster's released 2PP unless it appears the pollster is getting it wrong, in which case I will calculate my own 2PPs from their primaries.
House Effect Correction:
Poll types by a specific pollster are applied a house effect correction when it is considered that they are displaying a house effect in favour of a particular party. This can either be a systematic long-term feature of that poll series or can reflect a pollster developing a sustained medium-term difference from the tracking shown by other polls. The minimum house effect correction is 1 point, so if a pollster is suspected of having a house effect smaller than that, then that is treated under Global House Effect (see below).
The current house effects applied are:
AMR: +1 added to Coalition
Global House Effect Correction:
A global house effect correction of up to a point is applied to the aggregate if it is considered that the pollsters included in it (after the adjustments above) tend on average to lean to one side or the other. This is partly determined by considering the performance of polls at a range of election campaigns, with federal elections considered most important but state elections also taken into account. However short-term changes in house effect in specific polls are also taken into account. The frequency of the polls included is also taken into account - if polls that lean slightly to one party are more common or more heavily weighted than those leaning slightly to the other, then this is corrected for.
(Note that while primary votes for the polls that do not have house effect corrections applied tended to favour Labor in final 2013 polls (by about 1 point on a 2PP basis once 2013 election preferences are applied), (i) I suspect that up to 0.6 points of this effect can be explained by changes in the behaviour of the polls through the campaign, ie final polls by some pollsters were by chance more ALP-friendly than earlier (ii) there was probably late drift from Labor to Others.)
The current global house effect correction is +0.3 to Coalition.
Sample size:
No scaled adjustment is made for sample size at present (since most reliable national polls use broadly similar sample sizes which they increase at election time), except that a poll with a sample size of less than 950 automatically has its value in the aggregate halved.
Unusual polls:
From time to time a poll is available that includes data from around half the Australian voter base. Examples are a Galaxy poll of only female voters, and a ReachTEL poll of only NSW and Victorian voters. In these cases a national figure is estimated based on known differences between the surveyed and non-surveyed components, and the poll is aggregated at half its normal value. However such polls by pollsters without well-established national 2PP tracking (eg JWS Research bulk polls of dozens of "marginal seats") are not included. Single-state polls of federal voting intention are also not included.
New poll types:
A new poll type by an established pollster may be either included immediately, or excluded until three polls by that type have been conducted. The new poll type is always excluded until three polls by that type are conducted if it appears to have a substantial house effect.
Example tracking:
The following is the unsmoothed tracking graph for this aggregate as of Wednesday each week from 9 Oct 2013 to 11 Dec 2013 (I'm assuming no unexpected polls are released this week!)
Changes to the aggregate will be edited in above. Sections that become redundant will be removed. Issues requiring substantial discussion will be the subject of updates added below.
Seat Projections:
I may add a seat projection feature in the near future, based on probabilistic modelling . For the time being it should be noted that a narrow 2PP lead for the ALP, when it occurs, does not necessarily imply that the party would win an election "held now".
As a general rule if the Coalition is leading the Coalition would probably win an election "held now" (assuming the campaign did not change voting intention), if Labor is leading 51.5-48.5 or better than Labor would probably win, and if Labor has a narrow lead or the aggregate is tied then the outcome would depend on how votes were distributed, and might not be an outright win to either side.
Other Aggregators:
Currently operating 2PP aggregators include:
Bludgertrack (similar concept but very much fancier stats methods)
Mark the Ballot (ditto, assumes house effects sum to zero)
The Phantom Trend (similar to Bludgertrack, very statistically advanced. Usually leans to Coalition more than other aggregators presumably because of different assumptions about house effects.)
AFR Poll of Polls (very basic tracking and graphing of weekly averages)
Several aggregators operated during the 2013 election cycle that have yet to emerge from hibernation or that have since been discontinued.
Conditions for operation:
The aggregate is suspended (frozen at its existing value) at any time at which the number of individual polls included in it falls below three. When the number falls below five a cautionary note is added.
Mandatory reset events (all existing polls discarded and the aggregate restarts after three new polls) are (i) a federal election for the full House of Representatives is held (ii) there is a change of Prime Minister. Resets may be applied in other circumstances but this will only occur, if at all, with great reluctance!
Log of adjustment and methods changes:
The following lists all changes made to the pollster-specific and global house effects, and any other changes made.
24 Feb 2014: Morgan multi-mode (SMS/face) house effect reduced from +1.5 to +1 (skew to Labor).
6 May 2014: Galaxy multi-mode added with weighting 1.1 (edit: changed on 19 May to 1.15 as it is partly an established method). ReachTEL house affect applied (+1 to Coalition) - I suspect temporarily - since polls over last six months have had equivalent values to Morgan. Morgan multi-mode by last-election preferences (inc face-to-face) weighting raised from 0.8 to 0.9 owing to evidence of stability. Previous requirement for ceiling for weighting for poll that attracts house effect correction dropped.
2 June 2014: Ipsos I-view added.
16 June 2014: Morgan multi-mode (SMS/face) house effect returned to 1.5. As Morgan have issued a smaller sample-size weekly sample a weighting of 0.8 is used for such samples until we have seen more of their behaviour.
23 July 2014: ReachTEL house effect removed as last two polls have been below aggregate for Labor and average for ReachTEL over the term now leans ALP by less than half a point.
22 Sep 2014: 2PP figure for inclusion (polls other than ReachTEL) changed to be an average of the released 2PP and a derived 2PP. Prior to this change only the released 2PP was used for polls other than ReachTEL. See Wonk Central: What Do We Do With The Poll Rounding Problem? for reasons.
7 Feb 2015: Global house effect set at +0.1 to Labor. This is mainly caused by apparent but in my view still inconclusive evidence that Ipsos has a house effect.
1 Mar 2015: Global house effect set to zero and Ipsos house effect set at 1 point (skew to Coalition).
18 May 2015: Morgan house effect reduced to 1 point. Note that this applies after the correction for the implied 2PP based on the primaries - the house effect of the released 2PP is still likely to exceed 1 point.
5 July 2015: Ipsos house effect removed as trending estimate of it has fallen below half a point (after correction for primaries). Global house effect set at +0.1 to Labor instead. Also, new Galaxy-run Newspoll added. It attracts a +.1 bonus for being administered by Galaxy, but no continuity bonus as it is based on an untried combination of methods (robo+online).
9 Aug 2015: Global house effect set to zero as new Newspoll has so far displayed a small and not yet statistically conclusive ALP lean. Change backdated to start of new Newspoll meaning that all readings since new Newspoll started adjusted by 0.1 to Coalition.
24 Aug 2015: Global house effect set to +0.1 to Coalition as new Newspoll continues to display slight ALP lean and also based on considering the average weighting the various polls typically carry in the aggregate at any time.
14 Sep 2015: Mandatory reset - replacement of Prime Minister. Three polls taken entirely after the replacement are required for aggregate to restart.
5 Oct 2015: House effect on Morgan suspended as two-Turnbull era polls have behaved very differently to Abbott-era polls.
12 Oct 2015: Weighting for Morgan reduced by 0.2 points because of unpredictable house-effect behaviour and my concerns about the reliability of both SMS and face-to-face polling.
20 Oct 2015: Morgan house effect set: +2 added to Labor (poll currently estimated to skew to Coalition by at least 2 points)
23 Oct 2015: Added explicit condition of half value for polls with no released primary votes.
30 Nov 2015: Morgan house effect reduced to +1.5. This applies to the current poll but is not retrospective.
1 Dec 2015: Essential house effect added and set to +1.5 added to Coalition as poll has skewed severely to Labor in recent weeks. One point backdated to last week; this is otherwise not retrospective.
9 Feb 2016: Essential house effect changed and added to +2 to Coalition as poll has skewed to Labor by average 2.3 points in last nine polls. Morgan house effect reduced to +1 as Coalition lean post-Turnbull is reducing. Both changes applied to today's polls, not backdated.
23 Feb 2016: Essential house effect reduced to +1 to Coalition, backdated to last week.
21 March 2016: Morgan house effect removed. Not retrospective.
19 April 2016: Essential house effect removed, but global house effect reset to +0.2 to Coalition.
27 April 2016: Global house effect reset to +0.3 to Coalition.
3 May 2016: Essential house effect restored to +1 added to Coalition (poll skews to Labor), global house effect reduced to +0.1 to Coalition.
8 May 2016: Global house effect reset to +0.2 to Coalition.
7 June 2016: Essential house effect removed, global house effect reset to +0.3 to Coalition.
Have you considered adjusting for City vs Country...
ReplyDeleteThe figures dont get released very often, but the last newspoll quarterly showed no bias, in 2013 there was a 3% difference, which makes sense from policy issues that where present during that election period.