July 5, 2012

Informing Research Choices: Indicators and Judgment

The Expert Panel on Science Performance and Research Funding

Summary

Lack of a “one-size-fits all” solution to monitoring and measuring research performance is driving a growing international need to develop and implement new methods and, share best practices and lessons learned.

In 2010, the Minister of Industry, on behalf of the Natural Sciences and Engineering Research Council of Canada (NSERC), asked the CCA to examine the international practices and supporting evidence used to assess performance of research in the natural sciences and engineering disciplines.

Request a Briefing or Ask a Question Send a Request
Close

The Question

What do the scientific evidence and the approaches used by other funding agencies globally have to offer, in terms of performance indicators and related best practices in the context of research in the natural sciences and engineering, carried out at universities, colleges, and polytechnics?

Key Findings

Informing Research Choices: Indicators and Judgment drew from both international and national approaches for evaluating research performance. Overall, the assessment presents an important opportunity to address science and performance from a uniquely Canadian perspective.

After examining the available evidence, the CCA’s Expert Panel concluded that quantitative indicators should inform, rather than replace, expert judgment in science assessment for research funding allocation. The Panel also developed four guiding principles to support research funding agencies undertaking science assessment in support of budget allocation, they are: context matters; do not harm; transparency is critical; and expert judgment remains invaluable. The principles are expanded upon with the Panel’s report.

  • Many science indicators and assessment approaches are sufficiently robust to be used to assess science performance at the field level in natural sciences and engineering.
  • Quantitative indicators should inform, rather than replace, expert judgment in science assessment for research funding allocation.
  • International “best practices” offer limited insight into science indicator use and assessment strategies.
  • Mapping research funding allocation directly to quantitative indicators is far too simplistic, and is not a realistic strategy.
  • There is no compelling reason for certainty that past successes will lead to future successes or past failures to future failures. As a result, science indicators may not always provide a reliable guide to future prospects.

 

Expert Panel

The Expert Panel on Science Performance and Research Funding