The Australian Bureau of Statistics’s admission — of what many suspected — that recent jobs data have been flawed is a full-blown crisis in one of the country’s most important institutions. The ABS is relied on by policymakers, the community, business and the markets to provide an accurate snapshot of what is happening not merely economically but demographically, socially and culturally. Its monthly employment data is one of its flagship surveys, one that is critically important to Treasury, the RBA and financial markets. And it’s broken.

The ABS had:

“… concluded that the seasonal pattern previously evident for the July, August and September labour force estimates is not apparent in 2014. This assessment was made while preparing labour force estimates for September 2014 and relates to all seasonally adjusted labour force estimates other than the aggregate monthly hours worked series. As there is little evidence of seasonality in the July, August and September months for 2014, the ABS has decided that for these months the seasonal factors will be set to one (reflecting no seasonality).”

Critically, it was in July that the ABS changed the criteria for “looking for work” and the questionnaire that went with it. As Crikey reported at the time, the ABS didn’t run a control group in July or August to check to make sure the data it was getting from the change was accurate. It blames cost pressures for the decision not to run the control group, although ultimately this debacle is as much about the internal judgment of the ABS as it is about funding cuts. In its statement about the July changes, the ABS said:

“… there is no evidence that the introduction of the new active job search steps and the changing of two active steps to passive has had a significant impact on the estimates for unemployment and not in the labour force. In addition, changes in participation and the number of unemployed persons did not occur in a consistent manner across states and territories, males and females, and age groups. This further supports the conclusion that the new questionnaire had no systematic impact on the estimates of unemployed persons and persons not in the labour force.”

In hindsight, that looks like a remarkable misjudgement. At the time it looked wrong and was actively disputed by economists and policymakers, including some of the most senior in the country. Their concerns about the accuracy of the move have now proven to be right. The problem became painfully apparent during the preparation of the September numbers, released this morning. With seasonal adjustment set to 1 (i.e. no change), unemployment has been steady at 6% for July and August but rose to 6.1% in September seasonally adjusted, with a fall of nearly 30,000 jobs and a 0.2% fall in participation. In trend terms, unemployment remains at 6% and participation is unchanged.

“Hockey pointedly blamed Labor for leaving the ABC with ‘insufficient resources’ and pointed to ‘some structural issues that need to be addressed, including the fact that they haven’t got a Chief Statistician’. Well Joe, what’s been holding you up in appointing one?”

Without taking action, the ABS’ jobs figures would have been utterly implausible for September: it would have reported a staggering fall in employment of 172,000, and a decline in participation of a full 1 point — although the net result would have still been 6.1% unemployment. This was too much volatility — and too much continued volatility from previous months. As the ABS said this morning:

“The movements between July and August 2014 for the seasonally adjusted Australian employment series, especially part-time employment, were very large but again were not unprecedented. However, if the previously observed seasonal factors had been applied to September, there would have been an unprecedented movement in full-time employment. Cumulative evidence from these three months identified that maintaining the standard approach was not appropriate.”

So what happens now? Well, “trust us for a while longer” is the answer from the bureau this morning:

“The ABS will… initiate a review into the series to better understand why the series appear to have shifted away from their usual seasonal patterns. Although estimates for October have not traditionally exhibited strong seasonality (and therefore have seasonal factors close to one), it is not ideal to maintain the current treatment for an extended period. The treatment in October and future months will depend on the estimates produced, the continuation of the current investigations and the outcome of the review.”

That’s all but saying we will be flying blind for a while longer. For the time being, the ABS suggests we rely on the trend series.

The ABS has endured years of efficiency dividends under Labor, and that continued this year. The bureau’s former head, Brian Pink complained in an interview early this year that the efficiency cuts had cost it a cumulative figure loss of close to $120 million. There hasn’t been a permanent replacement for Pink since he left, with Jonathan Palmer continuing as acting Australian Statistician. This morning in Washington, Treasurer Joe Hockey said he wasn’t prepared to “write a blank cheque” for the ABS and that the government was considering options such as charging users to access data. Hockey pointedly blamed Labor for leaving the ABS with “insufficient resources” and pointed to “some structural issues that need to be addressed, including the fact that they haven’t got a Chief Statistician”. Well Joe, what’s been holding you up in appointing one?

Clearly Treasury — which has been distracted by its own succession issues with the political assassination of Martin Parkinson, the departure of David Gruen and having to sack hundreds of its own staff — alerted Hockey to the seriousness of the problems within its key portfolio agency, given Hockey has a cabinet submission in the works. And these issues have been years in the making. In 2008, after a $20 million budget cut in Wayne Swan’s first budget, the ABS decided to save money by reducing the size and the scope of the sample of both the retail trade and the labour force surveys. At a time when the GFC was breaking over the economy, the Reserve Bank, Treasury, banks and politicians were all but in the dark about how two of the main factors in the economy were travelling – household spending and jobs creation/destruction. For several months, the data produced was inconsistent, and the ABS was forced to revert to the older models.

And, finally, what would Ian Castles have thought? He was the Australian Statistician from 1986 to 1994 (which would have disqualified him for any role in government under the current government, especially given that Malcolm Fraser appointed him as Secretary of the Finance Department ). The late Castles, who would go on to help revamp Britain’s statistical collection and analysis and help free it from government interference, was in a line of Australian Statisticians who helped shape and give the ABS its reputation as one of the finest statistical agencies in the world.