ANAO: Home Affairs’ evaluation and reporting processes 'not fit for purpose'

The Department of Home Affairs continues to invite one reprimand after another, this time by giving the Australian National Audit Office ample reason to criticise its handling of advice to the government over a pilot visa program for skilled refugees.

Home Affairs’ evaluation and reporting process was so bad that the Audit Office labelled it “not fit for purpose”.

The department has responded, saying it is “actively seeking to mature” its evaluation culture.

Immigration Minister Andrew Giles expanded the Skilled Refugee Labour Agreement last year to 500 places on advice from the department.

The initial pilot, introduced by the former Coalition government in 2021, allowed for 50 to 100 places and was expanded to 200 in March 2022.

The scheme allows endorsed Australian businesses to sponsor skilled refugees through a specific agreement with the federal government and the issuing of specific visas.

Labor increased visa place numbers to 500 in July last year.

While the evaluation of other pilot programs tendered by the Health Department and the Department of Veterans’ Affairs essentially got ticks from the auditors, Home Affairs got a reprimand in the ANAO’s report this week.

“Home Affairs’ reporting of outputs of the Skilled Refugee Labour Agreement pilot did not contain fit-for-purpose analysis and does not satisfy the requirements of evaluation reporting in the Commonwealth Evaluation Toolkit,” the audit report states.

That evaluation toolkit lists a string of red dots to indicate where elements of the evaluation were not met in the advice the department gave to the minister.

They include:

• Evaluation questions are not identified, and there is no ‘methodology’ section.
• No sampling was conducted. To the extent that participants in the pilot are considered to be participants in the internal evaluation, all pilot participants are represented in the data reported, rather than data only being reported for a sample of participants.
• The number of participants involved, the types of visas granted and the number of days to process the visas are provided. Analysis of such data is limited to calculating averages.
• The ministerial brief includes data analysis by TBB [Talent Beyond Boundaries], including the average salary of pilot participants, retention rate and number of employers by sector. This was prepared by TBB, and there is no evidence of how Home Affairs gained assurance from the data provided.
• There is no consideration of ethics.
• Findings relate to specific implementation and design challenges. These findings are drawn from general observations of limited performance data rather than from an evaluation exercise. Findings are described along with implications for design choices to be made for the expanded pilot. These findings typically do not derive from data analysis of pilot outcomes, but rather qualitative discussions.
• Recommendations are made as to continuation of the pilot and changes to its design. The recommendations derive from limited qualitative analysis conducted by TBB rather than from an evaluation activity.
• No limitations are identified.

In a further scathing critique of the department’s slap-dash approach to informing the government, the audit office said: “Home Affairs’ advice to the Australian Government for the continuation of the SRLA pilot was not informed by an evaluation.”

And elsewhere in the report: “The Department of Home Affairs did not complete its planning for, or undertake, a robust evaluation for the Skilled Refugee Labour Agreement pilot.

“All evaluation plans and approaches could have been enhanced by a greater focus on the availability of data and an assessment of the proper use of public money.”

While the audit agrees that the ministerial brief contained “high-level data about program outputs, such as the number of visa applications and grants, and average processing time”, there was little explanation supporting the figures.

“Home Affairs did not undertake any data analysis or stakeholder engagement to understand how the experiences from the pilot could inform recommendations or validate the information provided by TBB,” it states.

The audit office concludes that Home Affairs’ evaluation of the pilot was, at best, only partially effective.

Responding to the National Audit Office’s report, Home Affairs Secretary Stephanie Foster accepted the findings.

“The department agrees with the findings and recommendations and is actively seeking to mature the evaluation culture and practices across the organisation,” she wrote.