Poll Bludger: how did the pollsters do?

With all but a few votes remaining to be counted for the lower house from the July 2 federal election, we now have a clear basis for assessing the performance of the opinion pollsters, who had scarcely less to prove at this election than the party leaders.

Scepticism of polling has been mounting internationally in recent years, owing to dramatically declining response rates and an industry-wide record of poor performance across much of Europe in recent years — most memorably in Britain last May, when the Conservative vote was uniformly underestimated by around 6%.

Suspicions that the Australian polling industry might be about to hit similar trouble were heightened by the recent demise of the Newspoll organisation, with The Australian now outsourcing the polling that appears under that name to Galaxy Research and international market research giant Nielsen’s local polling operation, which had been providing the Fairfax papers with a continuous poll series going back to the 1970s.

[Election deciders: meet the pollsters]

That left Australian polling largely in the hands of quite new outfits that in many cases plied novel trades like online panel surveys and automated phone polling.

The election presented a credibility test for the new aggregation of pollsters and poll methods individually, and of the very endeavour of opinion polling collectively.

So far as polling of national voting intention is concerned, the industry couldn’t have hoped for a better result.

The primary vote results of the last Newspoll of the campaign, which was conducted from a bumper sample of 4135 from the Tuesday to the Friday and published the night before polling day in The Australian, were Coalition 42%, Labor 35% and Greens 10% — exactly as the election result was after rounding to whole numbers.

A separate poll Galaxy Research conducted for the News Corporation tabloids a few days earlier was marred only to the extent of recording 11% for the Greens, whose primary vote currently stands at 10.3%.

A ReachTEL poll conducted two nights before polling day was only fractionally less accurate, while Essential Research and the Ipsos poll for Fairfax got the big things right, while landing a little too high for the Greens, particularly in the case of Ipsos.

However, opinion poll sceptics don’t lose out completely, thanks to widespread inaccuracy among polls conducted at electorate level.

Such polls have exploded in popularity over the last three electoral cycles, both with media outlets and an array of private clients, with results for the latter filtering through to the media increasingly often.

This is despite the fact that seat polling has a very shaky track record, which certainly wasn’t improved by their performance last month.

Most of the seat polls from the past campaign were conducted by ReachTEL or Galaxy from samples of around 600, with corresponding margins of error of around 4%.

A degree of imprecision can thus be expected, but as a statistical analysis on my blog shows, they behaved more like the margin of error was 7%, and with an overall skew of at least 1% in favour of the Coalition.

The underlying pattern is vividly illustrated by the following chart, which plots the error in favour of the Coalition on two-party preferred for each poll on the horizontal axis, and the swing to the Coalition in that electorate plotted along the vertical axis (so errors or swings in favour of Labor show up as negative).

The clear downward pattern shows that the bigger the swing to Labor, the worse a job the seat polls did of predicting it — or put another way, that results tended to look too much like the last election and too little like this one.

Two cases that illustrate the point are the seat of Macarthur at Sydney’s south-western fringe, where Liberal member Russell Matheson was dumped by a swing of 11.8%, and Tasmania, where the statewide swing was double the national result at 6% and three of the five seats flipped from Liberal to Labor.

[Seats to watch: storm-ravaged northern Tasmania, where the winds of change roar]

Macarthur was the subject of two Galaxy Research polls and one each for Newspoll and ReachTEL, all of which had the two-party vote either at 50-50 or very slightly in Labor’s favour.
None came anywhere near Labor’s ultimate winning margin of 8.4%.

The biggest and most surprising swing in Tasmania was the decisive 10.1% shift to Labor in Bass, a seat that was polled four times during the campaign — three times by ReachTEL and once by Galaxy as Newspoll — without any indication that the swing would exceed 3.5%.

Seat polling will probably remain popular with media outlets, as it meets the human interest requirement for news stories by putting names and faces to winners and losers.

But if local media outlets are to provide real enlightenment about the state of the electoral horse race, they would do better to zoom out from marginal seats and focus on polling at the statewide level.

That way they will have an easier time matching survey samples to the demographic characteristics of the overall population, and not encounter quite so many respondents who have been polled to death already.