October 19, 2020
Professor Sue Thomas
Chief Executive Officer
Australian Research Council
Dear Professor Thomas
Thank you for the opportunity to respond to the ARC’s consultation on the Review of Excellence in Research for Australia (ERA) and Engagement and Impact (EI) measures on behalf of the Group of Eight (Go8). I note that Go8 members may make individual submissions to the review including on technical and detailed operational aspects of both ERA and EI.
This Go8 submission is deliberately strategic and high-level representing the views of our members who win the majority of ARC grant funding – 68% of ARC income reported in 2018. Collectively in ERA 2018, Go8 members received a rating of world standard or above for 99.6% of evaluations at the 4-digit Field of Research (FoR) level. Go8 ratings at the very highest rating – well above world standard – were 55% of all Go8 evaluations and also represented 50% of Australia’s total ratings at this level.
As noted, the Go8 is taking a high-level approach to ERA and EI, looking to set them in the context of the overall national research and innovation effort. To that end, I would refer you to the Go8 research policy document Enabling Australia’s Economic Recovery Through Supporting Research Excellence. This document has been attached for reference.
The three key recommendations of that document are:
- Supporting excellent research at scale to maximise benefit to the nation;
- Sustainable translation and research infrastructure funding across national economic priority areas; and
- Transparent and full costing of research to ensure effective expenditure.
It is primarily in delivering on this first recommendation that ERA is a key tool. As noted in the Go8 research document, ERA is in fact “one of the world’s most respected research quality rating measures” and the Go8 believes part of a robust, transparent and regular assessment of university research.
Having set this context, the Go8 makes the following recommendations:
- That public reporting of ERA results includes both an excellence rating and a research volume (scale) indicator at the Unit of Evaluation level – that is institutional Field of Research level.
- That the ARC streamlines the research data collection so that the administrative burden for institutions for both ERA and EI is reduced. In particular, that the ARC explore ways in which publicly available data sets could be used – with appropriate controls and oversight – in the ERA assessment process.
- That ERA and EI move to an extended assessment cycle to allow time for evolution in institutional performance. The Go8 recommends a 5-year cycle with consideration given to off-setting ERA and EI assessments.
- That reporting of EI results and impact studies provides strong examples to the Government and public on the contribution of university research to economic growth and national priorities. This could include a greater emphasis in reporting on Socio-Economic Objectives (SEO) with EI results, the potential inclusion of identified national priorities in EI Impact Studies, or even consideration of EI Impact Studies being based on SEOs rather than FoRs as currently.
One final matter that I would raise for consideration is the proposed changes to the Provider Category Standards (PCS) that have been flagged by the Government. As you will be aware, the PCS as part of the Higher Education Standards Framework set out the requirements for institutions to be registered in a particular category of Higher Education providers.
In drafting new PCS, the Government has accepted the recommendation of the review conducted by Professor Peter Coaldrake AO in 2019 that enhanced research thresholds be specified for the Australian University category. To satisfy these thresholds (as currently drafted) institutions would need to undertake research at or above world class standard in at least three, or at least 50 per cent, of the broad (2-digit) Fields of Education it delivers, whichever is greater (from 1 January 2030). Legislation enabling these changes is currently under consideration by Parliament.
If ERA remains Australia’s principal assessment of the excellence of university research, then it will likely be a critical element in assessing the performance of universities against the PCS research thresholds. For this reason, it is important that the ARC consider this in the design of ERA and also conducts ongoing engagement with both the university sector and the Higher Education Standards Panel accordingly.
Detailed discussion
Research Excellence at Scale
As noted above the Go8 blueprint, Enabling Australia’s Economic Recovery Through Supporting Research Excellence[1], advocates that the Australian Government supports excellent research at scale to maximise benefit to the nation.
ERA demonstrates Australia’s research strengths in the areas that make a direct contribution to the delivery of public services, to meeting national priorities, building the future workforce and capability and developing policy. This goes to the heart of ensuring we support excellent research at scale to maximise benefit to the nation. In this context, the ERA exercise is a success, notwithstanding that the process can still be improved.
However, it is not just excellence that should be rewarded but excellence at scale – scale which demonstrates consistent capability within specific fields of research and across fields of research.
In this respect, the public reporting of ERA falls behind global counterparts such as the United Kingdom’s Research Excellence Framework (REF) that provide a full picture of research excellence by reporting both an excellence rating together with the volume of research being rated. This allows a proper public policy debate on research based on precise knowledge of the scale or excellence across all discipline areas in the UK university research system.
While Australia may not choose to replicate the UK system, it is critical both from a public transparency perspective and in order to permit a fully informed public policy discussion of supporting research that ERA publicly reports both excellence ratings and the volume of research under assessment.
Streamlined Data Collection
As previously noted by the Go8 in relation to ERA[2], and relevant to EI, assessment of research quality as well as research impact and engagement can be aided by data already collected by Government. Such datasets include the Higher Education Research Data Collection, the Australian Bureau of Statistics Research and Experimental Development, Higher Education Organisations, Australia (HERD) datasets, and the replacement for the National Survey of Research Commercialisation previously managed by the Department of Industry. Greater use of ORCID, DOI and bibliometric databases – with appropriate review by institutions – could also potentially reduce the submission burden on institutions.
Major efficiencies should be achievable if such data sets could be used in thoughtful and tailored ways, including if possible, to pre-populate institutional submissions against pre-determined units of assessment. The cost – estimated for ERA to be $1 million for a single major Go8 alone – and impost to universities may sharply fall if the ARC could work within its Education portfolio and others in a consent-driven way to ensure that such data is usefully shared. As the Minister for Government Services, Stuart Robert, noted in introducing the Exposure Draft of the Data Availability and Transparency Bill 2020 in September 2020 relating to public data, ‘we need to get better at using the information we already collect, instead of asking the same questions again and again’[3].
To ensure that integrity and accuracy of inputs are maintained, institutions should still have the option to review and change pre-populated or pre-assigned data as needed.
Extended Assessment Cycle for ERA and EI
As communicated to you in July 2019 in correspondence from Go8 DVCs-R Chair – Professor Duncan Ivison – the Go8 recommends that the interval between both ERA and EI rounds be lengthened to five years and hence supports that the next ERA exercise be held in 2023 as scheduled.
It is the view of the Go8 that an assessment cycle of every 3 years – as was the case between ERAs in 2012, 2015 and 2018 – is too narrow and too frequent. For instance, a 3-year cycle and a 6-year window of assessed research outputs means that many outputs are assessed twice. This promotes both a short-term approach to institutional management of the research portfolio – particularly in terms of strategic investment in developing research capabilities – and also fails to provide a time scale long enough to assess the evolution in performance of an institution.
Consideration could also be given to either alternating between ERA and EI in each 5-year cycle or offsetting ERA and EI in their respective 5-year cycles.
From an institutional perspective, these considerations will be strongly influenced by the perceived administrative burden of ERA/EI – as discussed above. This is reflected in discussions within the Go8 and may be reflected in the specifics of institutional submissions.
Effective use of EI Results in Evidencing the Contribution of Research to the Economy and National Priorities
While the EI process has been through only a single assessment round, the Go8 believes that it has not been completely successful in its first listed priority to provide clarity to the Government and Australian public about how their investments in university research translate into tangible benefits beyond academia.[4]
There is a need to demonstrate in a clear and straightforward manner to the Government and the Australian public, how research contributes directly to the Australian economy and jobs, national priorities, as well as to societal and community well-being. This needs to be backed by a robust evidentiary base linking to the excellent research underpinning the impact.
The Go8 believes that the failure of the first EI process to completely deliver this, while partially due to the learning curve that institutions are on with respect to articulating impact, is largely due to the design of the EI process and impact studies.
Of course, this is a complex design matter that cannot be quickly resolved. In the consultation for the development of the EI Guidelines in 2017 (and afterwards) there were detailed methodological discussions on the design of the EI and its implementation. In particular, the Go8 submission emphasised the value of using SEO codes for impact case studies as an outwards-facing indicator of where in the economy or Australian community/society research is having impact.
To that end, it is worth considering whether EI impact studies could have a stronger emphasis on SEO codes, and perhaps also a link to identified national priorities. One way to do this effectively would be by basing the selection of impact studies by SEO code rather than by FoR as is currently the case. The Go8 acknowledges that this would be a significant change for the design of the EI and as per discussions above would need to be done without introducing significant administrative burden for institutions.
Please do not hesitate to contact me at vicki.thomson@go8.edu.au or 0417 808 472 if you would like to discuss further any elements of the Go8 submission.
Yours sincerely
VICKI THOMSON
CHIEF EXECUTIVE
[1] https://go8.edu.au/wp-content/uploads/2020/09/Go8_Research-Excellence.pdf
[2] Go8 submission to House of Representatives 2018 Inquiry into Funding Australia’s Research
[3] Department of Prime Minister and Cabinet, Data Availability and Transparency Bill 2020 Exposure Draft Consultation Paper September 2020
[4] Australian Research Council—EI 2018 Submission Guidelines, Page 6