Implementation of an Evaluation Model for Evaluating Complex

Implementation of an Evaluation Model for Evaluating Complex Health Research Outcomes American Evaluation Association November 1 -5 2011 Kathryn Graham (Ph. D), Heidi Chorzempa (MSc)

Overview of Presentation • CAHS Implementation and Results • Lessons Learnt • Next Steps

Our Organization and Mandate • Alberta Innovates Health Solutions has a three decades long history of supporting medical/health research • “Support, for the economic and social well-being of Albertans, health research and innovation activities aligned to meet Government of Alberta priorities, including, without limitation, activities directed at the development and growth of the health sectors, the discovery of new knowledge and the application of that knowledge”.

Why We Evaluate Health Research? “…. . economic and social well-being…. Source: Canadian Academy of Health Sciences 2009

Implementation Approach and Considerations Organization’s Mission Inputs 1. • • • Evidence-Based Practice Environmental scans Theoretical & Conceptual Models (CAHS) Retrospective Data Mine Studies Prospective reporting Psychometric Testing 2. Practice-Based Evidence • Integated into the business model • Stakeholder/Actors need, values, use • Evaluation tools e. g. collaborative logic models 3. Top Down and or Bottom Up • Policy/Priority Driven and or Discovery Driven Results TIMING – SHORT TO LONG TERM Organizational and Health Research Inputs, Processes, Reach, Outputs & Outcomes Indicators & Metrics Socio. Economic Impact Categories Indicators & Metrics 4. Context • Strategy, External Factors, Clusters, Domains 5. Levels of Aggregation Macro, Meso, Micro 6. Target Audience • Tailor results to stakeholder needs & values Mixed Methodology Approach

Drivers for Retrospective Study • Verification: Using CAHS - does it make sense? Were the impact categories, sub categories and measures meaningful? • Applying Theory to Practice “in Context” - Alberta’s story • Evidence Informed Decision Making: Lessons learned informing design of new AIHS programs

Retrospective Data Mine Study 242 health researchers supported Program Duration Pillar I: Biomedical Research Pillar II: Clinical Research • Scholar • Scientist • Sr. Scholar • Clinical Investigator 5 years 3+3 years Pillar III: Health Services • Scholar • Scientist • Sr. Scholar 5 years Pillar IV: Population Health • Population Health Investigator 3+3 years

How we did it…. • A retrospective review of finalized grants from 2004 -2008 • Population of a database • Review 855 annual reports across 8 types of personnel grants • Recognizing project limitations: - Attribution and contribution - Self-report - Data variability - Outcome underestimation - Timing of measurement - Retrofitting

Retrofitting the data to the model Annual Report Template Advancing Knowledge Capacity Building chapters # trainees - Grad students - Post Docs - Summer students - Other trainees # abstracts * Infrastructure # publications # book # of invited talks * External PI funding amount Informing Decision Making # of collaborations # of influences in guidelines # of influences in policies Health Impact # of diagnostic/ prognosis advancements # of treatment advancements # of media citations # of access improvements # of influences in public education # of safety improvements Socioeconomic Impact # patents # spin-off companies # products in development

Impact Summary Research Diffusion 100% Research activity National/ Global Research 11 collaborations with Health Authorities 100% Alberta Innovates Health Solutions • Advancing Knowledge Capacity Building Informing Decision Making Health Status and function, well-being, economic conditions Health Impact Socioeconomic Impact Across 4 Research Pillars Research Results 15% Knowledge Pool • 3833 publications • 4398 abstracts • 386 book chapters • 2789 invited talks Research Capacity • $217, 381 in attracted funding • 990 trainees on average per award year • 14 new laboratories 1196 Collaborations with other Researchers 90% 21% That influences decision making in…. That produces results • 5 guidelines 25 collaborations with Industries • 17 products developed 17 collaborations with Government That affects healthcare and health risk factors Health Care • 42 improvements to health care effectiveness • 8 improvements to health care efficiency & accessibility Improvements in: • 30 therapeutics • 12 diagnosis/ • prognosis techniques • 4 policies • 3 data services 18 collaborations with Non-profits • 8 guidelines That contribute to changing health, social and economic wellbeing Improvements in health and wellbeing • Qualitative Results Economic & Social Impact Determinants of Health • Personal behaviour • Social/cultural determinants • Environmental • 107 patents • 10 products in development • 5 spin-off companies The Public • 16 media events Impacts feed back into inputs for future research

Timeline of Innovation 2001 -2008 “I have designed a new approach where a target gene can be cloned into a large number of expression vectors by combinatorial assembly of the target gene with a limited number of pre-made DNA fragments. ” Scholar, year 2 Significance: II created a procedure that can disperse many copies of one gene with limited resources. This could be used to increase research productivity (for example, placing the gene responsible for insulin in many different cell types to try to determine the cell that produces insulin the most efficiently. 2001 “We were the first lab in the world to describe the functional consequences of mutations in T-type channels linked to childhood absence epilepsy in humans” Sr. Scholar, year 3 Significance: II found differences in the calcium channels in epileptic patients compared to non-epileptics. This could elucidate the pathophysiology of epilepsy as well as potential therapeutic venues. 2002 “The second paper in J Neurophysiol was cited by the Faculty of 1000 Biology as highly innovative. This paper explained why thalamic stimulation is not broadcast widely to the motor cortex and why DBS does not disrupt motor control. It also helped explain the mechanism of action of subcortical DBS for severe refractory depression. “ CI, year 4 Significance: Helps explain how deep brain stimulation works, which could lead to improvements on this treatment modality. 2003 2004 2005 2006 2007 “This lab contains state-of-the-art computing infrastructure… and likely the first in a medical imaging lab in Canada)” Bio. Scholar, year 2. “The data in this manuscript were the first to show that isoprostanes have the ability to modulate inflammatory processes with relevance to airway respiratory diseases. “ Sr. Scholar, year 3 Significance: II has cutting edge technology which could be used to improve diagnostic/prognostic modalities or be used for research (higher resolution images may display more information that is needed for progression). Significance: II found evidence for the use of isoprostanes (a form of anti-inflammatories) in the treatment of conditions such as asthma and COPD, where inflammation is the main source of disease progression. “The former was developed in collaboration with _____& _____and represents the first imaging tool to assess disease activity in both spine and sacroiliac joints of patients with AS. ” Sr. Scholar, year 2 Significance: II developed a tool that can be used to assess progression of anklosying spondylitis (a chronic and painful condition). This can help determine guidelines for treatment (ex. When to use a specific therapy) *Please note that this timeline only represents a small proportion of the 320 innovation statements captured during the review. “In addition, these studies showed for the first time, that chimerism induces a dominant form of humoral tolerance, allowing B cell tolerance to multiple donors. ” Bio. Scholar, year 5 Significance: II showed that a combined protein can improve tolerance of donated pancreatic tissue, which could improve long-term success rates of islet transplantation in diabetics. 2008 “In 2007, we published the first major report from the Alberta Diabetes Surveillance System (ADSS), in collaboration with the Health Strategies and Surveillance Branches of Alberta Health and Wellness (AHW). ” H. Scholar, year 4 Significance: The increased surveillance abilities resulting from ADSS will help inform health authorities’ efforts to reduce the burden of diabetes on Albertans.

© Performance. Measurement. ca What we did next Retrospective Data Mining Studies • Applied to Trainees Program Prospective Pilot Studies Science of Science • Collaborative Logic • Partnered in a Models science of • Created pilot science network reports for 3 programs • Collaborative Balanced Scorecard at the organizational Level Integrated Program Design • New program design • Link to Goals/Objectives and reach • Psychometric testing for key inidicators

Example of the Balanced Scorecard Strategy Map Courtesy PM 2 Linking and Rating Processes Scorecard Information – Indicators Scorecard Data – Dash board Linking and Ranking Projects

Lessons Learnt • Verification – “YES WE COULD DO IT” • - Theory to Practice Lessons – Implementation Complemented CAHS model with other models Imbed knowledge translation – decision making rubrics – progress markers Link collaborative activity to outcomes Tell the story through qualitative and quantitative indicators • - Context Lessons Alberta Innovates System – innovation Link organizational and program goals and objectives to CAHS categories Tease out Innovation • Challenges • Change Management – theory of change • Utilization – is this a useful model for evaluating health research funding

Values and Valuing of Measurement “We are noticing a shift around the measurement of scientific value versus social/public value as well as the concept of collective action and shared value. Using a staged implementation approach, the CAHS model provided us with a best practice framework to measure benefits across the spectrum.

Contact Information Kathryn Graham (Ph. D) Director, Performance Management Alberta Innovates Health Solutions 1500 – 10104 103 Avenue N. W. Edmonton, Alberta, Canada T 5 J 4 A 7 Email: kathryn. graham@albertainnovates. ca Heidi Chorzempa (MSc) Manager, Performance Management Email: heidi. chorzemp@albertainnovates. ca THANK YOU