The CIHR Perspective on Impact Measurement Presentation to

The CIHR Perspective on Impact Measurement Presentation to International Healthcare Funders March 30, 2015 David Peckham Chief Audit and Evaluation Executive Canadian Institutes of Health Research

Introduction • The collection of attributable output and outcome data is challenging and can require the use of considerable resources – funding organizations struggle with finding feasible and sustainable methods for measuring and tracking the impact of their investments in supporting health research

CAHS Impact Framework Canadian Academy of Health Sciences Report Making an Impact (2009) identifies 5 impact categories Advancing Knowledge Capacity Building Informing decisionmaking Health Benefits Broad Economic and Social Benefits

CIHR Performance Measurement Regime • ‘Toolbox’ of around 80 measures available (down from several hundred) • Based on CAHS Framework impact categories • Every CIHR funding initiative required to have a performance measurement strategy • Flexibility – comparability but also some latitude

CIHR Toolbox Impact Category Descriptions Impact Category Description Sample of Indicators Advancing knowledge This category includes discoveries/breakthroughs, contributions to the scientific literature (grey or peerreviewed) and may include measures of research quality, activity, outreach and structure. • Building capacity This category includes the development and enhancement of research skills in individuals and teams, additional research-activity funding (e. g. , funding partnerships) as well as the development/enhancement of platforms. • Informing decisionmaking This category includes the impacts of • health research in the areas of health related decision making (i. e. , • science/research, general public, clinical and managerial decision-making, practice and policy and health products decision making). • • Number of scientific products produced through CIHR funded research Number of reports/technical reports produce by CIHR-funded grants Number of research collaborations among CIHR funded grants and awards Total number of CIHR grants supporting teams Percentage of CIHR grants reporting citation by others Field analysis of citations 5

CIHR Toolbox - Impact Category Descriptions Impact Category Description Sample of Indicators Health impacts This category includes advances in the prevention, diagnosis, treatment and palliation as well as changing health status and determinants of health. • • Potential years of life lost in Canada Community well-being index Health system impacts This category includes advances and efficiencies in the way the health system functions. • Mortality rate from treatable causes in Canada (age-standardized per 100, 000 population) Hospital readmission rates in Canada Broad economic and social impacts This category includes commercialization of research discoveries, human capital gains, health benefits (specific costs of implementing research findings in the broad health system), well-being and social benefit indicators. • • • Total number of staff being paid through CIHR grants Economic productivity levels of sick people in Canada 6

Examples of CIHR Baseline Data Measure Baseline Data 2012 -13 Percentage of researchers reporting the 92% creation of new knowledge Number of applications received 18, 802 Percentage of fundable applications Co-author Analysis Actual Data 2013 -14 Trend 92% ↔ 16, 851 ↓ ↔ ↔ 69% • 13, 030 • 13, 581 international collaborations (49% collaboration (51% collaboration rate); rate) • 3370 • 3, 437 interprovincial collaborations (12% collaboration (13% collaboration rate); rate) • 8535 • 8, 959 intersectoral collaborations (32% collaboration (33% collaboration rate) 7

Examples of CIHR Baseline Data Indicator Ratio of leveraged-funds for Institute. Driven Initiatives Ratio of leverage-funds for Horizontal Initiatives Canada's international ranking for the number of health researchers per thousand workforce Impact Category BC Baseline 2012 -13 $1: $0. 36 Actual 2013 -14 $1: $0. 42 BC $1 : $0. 56 $1: $0. 76 BC 3 rd among External Data G 7 source available every 5 years 89% 94% Percentage of Institute-Driven grants reporting stakeholder involvement in the research process IDM Percentage of operating grants reporting translation of knowledge IDM 57% 61% Percentage of priority-driven grants reporting translation of knowledge IDM N/A 68% 8

Lessons Learned after 12 months 1. Measures in the Broad Social and Economic Benefits and Health System Impact Categories have less easily available data and there are concerns over attribution 2. Data availability can be an issue if relying on external sources – e. g. national statistical offices 3. Further streamlining may be possible 9