Who evaluates the evaluators?

This is a story about evaluation, the most important — and least sexy — story in public policy.

Evaluation doesn’t set pulses racing. It’s for people who can’t cope with the excitement of auditing. I should know, I worked in an evaluation section as a junior bureaucrat nearly a quarter century ago.

But it’s crucial to effective government, and we’re not doing it well. Barely at all, in fact. A lack of evaluation, for example, is a recurring theme in one of the greatest policy failures of Australian governments, Indigenous health. Time and again, academics, sector experts, reviewers and even politicians like Ken Wyatt, the current Indigenous Health minister, have lamented the lack of evaluation of Indigenous health programs in order to better target them.

Labor’s Andrew Leigh has just announced the biggest development in evaluation in decades: Labor will set up an “evaluator general” within Treasury, “to conduct high-quality evaluations, preferably randomised trials, of government programs.” Leigh’s big on randomised public policy trials — the bibliorrheic economist recently wrote another book, Randomistas, on random trials in public policy — and the evaluator general, with a $5 million budget, will be point person for this new approach in Australia.

The key question, however, is whether Leigh’s colleagues will actually want to know whether their policies work or not. 

Evaluation used to be big in the Commonwealth, back in the 1990s. Public service departments had dedicated sections whose role was to select programs and evaluate whether they had achieved their goals. Departments were supposed to have evaluation plans, which detailed their evaluation work, and how programs would be designed from the outset so that they could be assessed for efficiency and effectiveness. But after the Howard government came to power, evaluation was kicked to the curb and partly replaced by the outcomes/outputs budget framework adopted at the turn of the century.

That meant — theoretically — each area of expenditure would have identified output indicators in the budget to enable assessment of how effective programs were. In reality, most of the indicators were process-related ones like “x million spent” and “95% of clients handled within specified timeframe” rather than anything about what happened in the real world. That left the evaluation space empty, at least until the Australian National Audit Office decided last year to expand its focus to program effectiveness.

The abandonment of evaluation was discussed by the National Commission of Audit in 2014, and it must have made for some uncomfortable reading for the Coalition.

A joint evaluation model, involving central and line agencies, operated in Australia between 1987 and 1997. Under this process there was a formal requirement for all programmes to be evaluated every three to five years. Each portfolio was required to prepare an annual portfolio evaluation plan and all new policy proposals needed to include a statement regarding arrangements for future evaluations. The process was also intended to provide formal evidence of programme managers’ oversight and management of resources. The model was considered reasonably successful but had shortcomings. As well as being resource intensive, many agencies regarded it as an external impost rather than a tool to improve policy-making.

Tony Shepherd and co — who elsewhere commented on the lack of evaluation of Indigenous programs — also put their finger on the bigger problem.

Ultimately, the success of an evaluation process depends on the appetite of ministers for rigorous assessments of programme [sic] effectiveness, and, importantly, their willingness to act on results.

There are no upsides to rigorous assessments of programs. As Leigh pointed out in his speech announcing the Evaluator General, the more rigorous an evaluation is, the more likely it is to find that a program hasn’t worked. What do you do with such an evaluation? Use it to cancel a program? Ministers hate that. It’ll result in a bad headline no matter how useless a program is. And if a program is successful, no one except policy wonks is interested — and certainly not the media. 

Shepherd and friends recommended forcing ministers to evaluate programs by requiring them to report back every budget process with the results of program evaluations, so that the results would feed into the budget process. And if they didn’t comply, Treasury and Finance could be sent in to vet the entire portfolio’s expenditure.

Leigh’s proposal is a welcome step in an important area. But if Labor is elected, he’ll need some compulsion to force his colleagues to cooperate.