Randomised testing good for policy development, says Leigh

If we tested new ideas the same way we test new pharmaceuticals, we’d be a better nation.

That’s the opinion of Andrew Leigh, assistant minister for competition, charities, treasury and employment.

In an address to the National Press Club on Tuesday (29 August), Dr Leigh helped launch the Australian Centre for Evaluation saying policy development in the public service should undergo more rigorous evaluation and include randomised testing.

At times talking directly to public servants, the Federal Member for Fenner suggested change in approaches to policy development could be driven by them.

“Public servants can become better consumers of evidence,” he said.

“When claims are made about the effectiveness of a program, ask about the quality of that evidence.

“Is it a single before-after study, or a systematic review of multiple randomised trials?

“Each of us can work to raise the evidence bar.

“Public servants can also help produce evidence about what works. If you spot an opportunity to run a high-quality evaluation in your agency, I encourage you to engage with the Australian Centre for Evaluation.”

Announced in the 2023 Federal Budget, the centre was given $10 million over four years and an initial 14 staff to work across government agencies to improve evaluation capabilities, practices and culture.

“A core role for the centre will be to champion high-quality impact evaluations, such as randomised policy trials,” Dr Leigh said.

“Past reports have clearly shown the need to improve the quality of evaluation across government.

“Work done for the Thodey Review of the public sector found that the quality of evaluation was ‘piecemeal’.

“Some high-quality evaluations have been conducted, including by the behavioural economics team in the Department of Prime Minister and Cabinet.

“But in many other areas, the capacity to conduct rigorous evaluation is lacking.”

Dr Leigh cited a report from the Australian Evaluation Society estimating that in 2021-22, the Commonwealth procured 224 evaluations from external consultants at a cost of $52 million.

“One problem for consultants is that there isn’t much incentive to undertake a high-quality evaluation,” he said.

“… the better that consultants design their evaluation, the less likely they are to produce a report that shows the program worked. Which may make it harder for them to win the next contract.

“That’s why we’re also encouraging agencies to rebuild their own in-house evaluation capabilities and consider partnering with the Australian Centre for Evaluation…

“Another reason that consultants’ evaluations may fall short is if they are commissioned to produce evaluations late in the process when there is insufficient planning and data available.

“So the Australian Centre for Evaluation will also be working with government agencies to strengthen evaluation planning, especially in new budget proposals, and ensure that evaluation is considered at all stages of policy and not seen as an afterthought.”

Dr Leigh said embedding evaluation in the work of government could take many forms over time – and not just running pilot studies.

He said when policies were rolled out to different sites over time, randomisation could be built into that rollout which would guarantee a rigorous evaluation. When funds were being allocated across states and territories, it would be possible to provide resources for those jurisdictions willing to conduct rigorous experiments.

He said more support should be given to programs backed by the best evidence.

“In the face of major challenges, low-quality evaluation is a hindrance, not a help.

“Using dodgy impact evaluation techniques is like doing your running training with a slow watch. It might make you feel like you’re fleet-footed, but when it comes to race day, you’ll eventually be shown up.

“That’s why researchers in areas such as pharmaceutical development are committed to using randomised trials.

“They recognise the importance of accurately evaluating new treatments. They know poor evaluation of medical treatments can cost money and lives.”

Dr Leigh said the Australian Centre for Evaluation would seek to take the same approach to policy – testing new ideas with the same methods used to test new pharmaceuticals.

“In the face of hard problems, we must bring more than a crash-or-crash-through mentality. We need to show up with a willingness to rigorously evaluate those solutions,” he said.

“We need to bring enough modesty to the task to acknowledge that answers which sound right may not always work in the real world.

“To generate and sustain a culture of continual learning, we need to be open to being proven wrong, and to use that information to do better the next time.

“We need to accept honest feedback – not pretend to get by with a dodgy wristwatch.”