March 29, 2019
Thank you for the opportunity to provide additional input to the Innovation Metrics Review by response to the ‘Improving Innovation Indicators’ Consultation paper released in March 2019.
The Go8 also thanks the Review for the previous opportunities to discuss innovation metrics in detail through the Go8 Directorate’s discussion with and input to the Taskforce, my follow-up letter to Dr Finkel (both in December 2018) and a meeting between the Taskforce and the Go8 Innovation and Commercialisation Group in February 2019.
This response reflects in part those discussions and is in addition to any submissions that may be made by individual Go8 institutions.
General
The Go8 notes that measurement data and the metrics that underpin them cannot be developed in isolation from the uses to which they would be put, including to inform government policy and drive behaviour by universities, industry and others. The Go8 emphasises the need at all times of careful and precise development and implementation of metrics including how data is categorised and understood for the purposes of collection – given how potent these measurements can be as a tool for advancing or disrupting innovation.
Overall, the Go8 supports the thrust of the Review expressed in the underlying principles especially the consideration of innovation wherever sourced and an openness to finding hidden innovation, including that which arises from creativity and not solely from research and development (R&D). While agreeing that it would be helpful to have specific criteria for deciding metrics for inclusion in the summary scorecard and the full suite of metrics, the Go8 notes that ultimately the chosen set of metrics in either case should be purposed to provide as accurate a view of the state of Australian innovation as possible.
A focus on metrics of themselves is insufficient. Whether data is available, what data is seen as relevant, how the metric is interpreted and what parameters are applied when responding to measurement surveys are some of the key issues that are encountered by universities and other respondents. There is a real question of how consistency of response affects the validity of the final measurement – anecdotally, this is encountered in a range of Government data collection exercises to which Go8 institutions are required to respond.
The Go8 notes that the key messages in Consultation Paper collectively appear to indicate that innovation is seen as serving primarily the customer, client, next-user or end-user, especially given the emphasis in remarks on benefits or effects for firms. The Go8 would emphasise the necessity in having an appropriate balance in measuring and understanding innovation across the spectrum – from the point of creation or idea-generation (including in formal research) to skill acquisition (including for future workforce) to technology adopters and adapters as well as solutions recipients (including but not limited to firms).
Acknowledging the many messages received to date by the Innovation Metrics Review and the potentially wide-ranging remit of the Review, the Go8 notes specifically the following for consideration.
Inputs versus outputs
There is some commentary in the paper that suggests innovation metrics focus too strongly on inputs to or activities of the innovation system rather than impacts. To an extent this may reflect what is perceived as being more easily quantitatively measured or time-defined (such as higher education expenditure on R&D, proportion of applied research conducted, numbers of researchers, numbers of invention disclosures) versus the much more difficult task of measuring impacts. Impacts are harder to categorise or count or relate directly or specifically back to the input[1] (such as the number of deaths prevented due to more accurate climatic prediction models based on cutting edge research) and are in many ways not time constrained (an innovation such as the eradication of a certain disease continues to impact throughout history until it re-emerges). Impacts are much more difficult to measure if based on actual effects imperfectly categorised or understood.
This is akin to a paradox that universities and researchers face – that research across the spectrum from fundamental to experimental is interconnected and that an over-emphasis (e.g. due to funding trends) on one location of that spectrum can compromise the entire research system at a point in time and well into the future. While an emphasis on impacts would seem logical, in that innovation is not just conducted for its own sake but to improve the human experience and so on, the innovation system similarly to the research system depends on effective flows (e.g. of knowledge), interconnections (where ideas may be sparked or charged by specific conversations between actors in different parts of the system), resource pooling (as in precinct activity) and cannot be simply defined by end results.
It is possible that no perfect form of measurement has been reached even among international counterparts. The Review provides an opportunity for novel metrics or indices to be developed domestically in recognition of these issues and must also recognise the necessity and benefits of contextualising the collected data (for instance country, economic, sector) trends for those who would use the collected data. Where other relevant data or evaluations exist, these should be effectively linked to create a more comprehensive picture. For example, the Australian Research Council’s Engagement and Impact initiative will potentially usefully provide qualitative context to the data on Australian research as a contributor to and facet of innovation.
Go8 recommendations
That the Innovation Metrics Review in its consideration of new metrics or a new approach to measuring innovation place equal and appropriate emphasis on all aspects of the innovation system including ‘inputs’, ‘outputs’ and the wide canvas of interaction ‘in between’ – drawing on but not limited to international methodological exemplars that can be translated in the Australian context.
That further emphasis be placed on how datasets can be better linked to each other and to other evaluation findings to provide a fuller and more accurate presentation of innovation trends including pathways to innovation.
That the Review recognise that fundamental or basic research as both an innovation and as a contributor to research needs to be better understood and that metrics to better explain its role need to be explicitly considered and developed.
Collaboration metrics and data
The Go8 strongly concurs with concerns expressed regarding the quality of data underpinning collaboration metrics and the need to have trusted collaboration data.
Specifically, this raises an issue in terms of the over-reliance historically by Government policy on OECD collaboration data. When reflecting Australia business-research collaboration performance is hindered by a difference in reference and reporting periods, the different approaches here and among Australia’s competitors in how the domestic surveys feeding data into the OECD collection are conducted[2] and (as noted in the discussion paper) discrepancy regarding what activities / sources are considered innovation, the Government’s continual reliance on Australia’s reported low levels of business-researcher and business-university collaborations as the basis to ushering in key policy changes is questionable. Indeed, it is likely that data on how effectively and precisely specific Government policies and programs have assisted in achieving innovation goals would usefully inform what adjustments, changes or enhancements to the innovation ecosystem could be made.
Regardless of the specific differences in domestic and international methodologies, Australia’s SME focussed business sector, with 99.8 per cent of businesses having less than 200 employees and 90.8 per cent having less than 20 employees[3] must have an impact regarding the degree to which collaboration is possible for less resourced organisations. As the OECD itself has noted, ‘collaboration with higher education or public research institutions constitutes an important source of knowledge transfer for large firms. In most countries, such firms are usually two to three times more likely to engage in this type of collaboration than small and medium-sized enterprises (SMEs)’[4]. Nevertheless, university collaboration with SMEs specifically for the purposes of innovation does and can occur as discussed recently at the Go8-BHERT Summit: Designing a Powerful New Industry-University Partnership in Australia. Particular effort should be taken to capturing those collaborations when and where they occur, potentially with a finer lens that might be applied to collecting data on collaboration between larger firms and universities.
Of specific interest would be how these collaborations form, as a foundation to assisting and demonstrating to other SMEs and universities how they can succeed at forming such partnerships. Data that is already collected and held by Government but not transparently available such as the usage trends of the R&D Tax Incentive and where specifically it drives collaborative activity would be useful for university policy-makers and practitioners to be able to access, as the data would inform where efforts could be focused and emphasised. The Go8 Innovation and Commercialisation Group has commented too on the usefulness of the National Survey of Research Commercialisation (NSRC) in providing an overall sense of trends even while some of the underpinning detail may be faulty and requires remedying. The Go8 would also emphasise the necessity of better context – including better definitions and examples where useful – to clarify what data is sought when the precise meanings of terms may otherwise be ambiguous.
Go8 recommendation
That the Innovation Metrics Review place appropriate emphasis on fine-tuning how collaboration is measured and understood in the Australian context, including by encouraging appropriate use of existing data sets collected by the Government including but not limited to the R&D Tax Incentive usage data. The Go8 also supports further refinement of the National Survey of Research Commercialisation especially as regards definitional and parameter issues with the survey to make it a more reliable and comprehensive dataset that appropriately reflects certain types of collaboration it measures. In all instances, the Go8’s interest is in how the measurements can assist in guiding what policy and practical refinements can be made – data collected must not only be transparently available but also meaningfully used.
Absorption capacity
Related to a degree to Australia’s business sector composition is the issue of how ready Australia’s economy is to absorb native innovation versus adopting and adapting internationally developed innovations. Further data regarding whether there is a significant issue in Australia in terms of absorptive capacity, what is affecting or influencing uptake of home-made innovation and ideas, and how this compares to Australian use of international innovations would assist in further understanding the limits or potential for uptake of Australian ideas, research, technological and social innovations. The Go8 is specifically interested in further examination of how and where university research and innovation is relevant but may not be readily taken up by potential adoptees. Conversely, how Australian innovation is adopted by overseas stakeholders should also be explored.
Go8 recommendation
That the Review recommend and focus as a subset on the development of metrics and data that can uncover the issues, motivations and barriers regarding Australian uptake of local versus international innovation, including capacity and time for industries and others to adopt new-to-world local innovation, such as university research that can be translated and commercialised.
Research infrastructure as a hub for innovation
The Go8 notes and supports comments in the paper advocating access to program or administrative data related to the National Collaborative Research Infrastructure Strategy (NCRIS) facilities and that innovation activity centring around research infrastructure facilities should be better measured.
As lead agents for around half of the NCRIS facilities, the Go8 collectively can fully appreciate how research infrastructure at that national level can drive innovation including via the necessity to collaborate when a resource is highly in demand, yet expensive and not easily duplicated. Such collaboration around the facility itself can spill over into collaborations over projects that use the infrastructure in ways that are not always predictable or expected to create innovative solutions. Yet this function or benefit of research infrastructure is insufficiently measured and understood, even while entire innovation precincts have been centred on major research infrastructure facilities. For instance it is perhaps insufficiently understood how research infrastructure can support research engagement with SMEs as a key subset of industry and how its physical absence from some locations impact on the degree of collaboration focused on innovation. Multiple examples of how innovation collaboration that arises around facilities available to researchers and users across the nation.
Go8 recommendation
That the Review consider the value of research infrastructure in focussing and driving innovation, and potentially recommend the development of metrics to measure its role in this regard. These could include though not be limited to: the instances of innovative solutions and practices that result for government and industry; research-industry partnerships that develop beyond the mere use of industry of the research infrastructure; the establishment of location-based precincts around research infrastructure facilities; the reliance of major global innovation on specific Australian facilities; and that specific Australian research infrastructure can in and of themselves be technological innovations.
Publication citations
The Go8 concurs with messages in the paper that advocate more precision in how publication citations are represented in metrics, including that further context helpfully informs what a researcher’s or institution’s publication record may mean in relation to innovation. The Go8 notes that existing methodologies and tools are employed to ‘normalise’ by field or discipline how citations numbers are considered and presented.
Go8 recommendation
That the Review consider and make recommendations around how publication citations data can be better employed and represented to advise how, where and to what degree innovation is occurring.
In closing, I would note that there are numerous key messages in the paper that the Go8 would support, such as the call for a focus on managerial quality and workforce skills and on understanding the flow of innovation (via technological, knowledge and talent transfer). The above input is intended to emphasise our chief areas of interest recognising that much ground has been covered in the paper.
Yours sincerely,
VICKI THOMSON
CHIEF EXECUTIVE
[1] This difficulty has been indicated in recent years by the degree of development that went into creating the Government’s Engagement and Impact measure due to imminently release its first-round results.
[2] The methodological issues have been discussed to an extent in iterative Australian Innovation Systems Reports released by the Department of Industry, Innovation and Science.
[3] Australian Bureau of Statistics (ABS) 2018, Counts of Australian Businesses
[4] OECD 2017, OECD Science, Technology and Industry Scoreboard 2017