Cochrane Rapid Response Accelerating Knowledge Translation for DecisionMakers

Cochrane Rapid Response: Accelerating Knowledge Translation for Decision-Makers Chantelle Garritty Ottawa Hospital Research Institute and Catherine Gallagher George Mason University

Overview Knowledge translation Evidence synthesis and knowledge translation: inseparable Decision-making dilemmas Requirement of high rigor evidence Lack of time Multiple restrictions Fallacies Cochrane’s approach Rapid Response: Borrowed heavily from KTA Canada Impact on translation Decision-making outcomes Rapid response v. Traditional systematic review Trades Impacts Evaluation

Rapid reviews in context…and as a precursor to translation Evidence needs arise… • Both organically and purposively • Within a system with static and dynamic operational parameters • At any level: Locally, nationally and globally • Always within limited resource structures Rapid reviews should… • Make the very best use of limited resources • Provide a gauge of the distance between status quo and optimum interventions or systems • Have clear implications for practice and for outcomes • Lead to implementation and assessment of impact Rapid reviews align perfectly with “habits of high value health care organizations”… • Specification and planning – criteria based decision making for clinical and operations • Deliberate design and management of resources for all systems and for sub-population • Measurement and oversight • Self-study ( Bohmer, 2011 NEJM) Here we discuss tools and methods to… • Enhance, automate, and accelerate processes • Limit biases of narrative input, increase transparency, and facilitate investment • Place the rapid review in its larger context of clinical and systems utility • More specifically , understand the relationship between acceleration and dissemination *Recognize the limitations of current depth and breadth of evidence pool – Bohmer, 2011 NEJM

Part I: Knowledge translation What is it? CIHR (2000) “the exchange, synthesis and ethically sound application of knowledge – within a complex system of interactions among researchers and users – to accelerate the capture of the benefits of research for the public through improved health, more effective services and products, and a strengthened health care system”

Emerged alongside Evidence-based Practice Movement: “process of moving what we learned through research to the actual applications… …. in a variety of practice settings and circumstances…. . . the interest in KT appears to coincide with the growing engagement in the EBP approach in which… …practitioners make practice decisions based on the integration of the research evidence with clinical expertise and the patient’s unique values and circumstances. ” Straus, Richardson, Glasziou, & Haynes, 2005

Coinciding with knowledge translation movement Increased: • rigor and standardization of all elements of the scientific process • burden of proof of utility of investigation • investigation into communication methods (dissemination sciences) • open access tools to summarize knowledge base, manage references • move publicly funded research to the public • stakeholder involvement • joint decision making It is impossible to separate knowledge translation from knowledge generation and synthesis

Why we need evidence synthesis and knowledge translation 1. To make sense of knowledge proliferation – 10, 000 new trials each year in MEDLINE – 350, 000 new trials each year in Cochrane 2. Public is not benefiting from evidence 30 -40% of patients do not receive care according to present scientific evidence 20 -25% of care provided is not needed or potentially harmful Grol & Grimshaw (2003) Lancet 3. Current publication practices do not speak language or format of decision makers (both societal and individual)

Why we need evidence synthesis and knowledge translation 4. Calibrate traditional academic model to meet practice needs Investigator interest versus Timeliness versus Simplicity of design versus Generalizability versus Academic freedom versus Increases in standardization versus Quantity of publications versus Social utility Exhaustiveness Complexity of social world Local acceptability Costs to public Loss of natural variation Quality of reconciliation

What does it look like? There are many models of the KT process Commonalities across models include: 1. Early and consistent stakeholder engagement 2. Standardization of reporting requirements to maximize benefits of research efforts 3. Concern with content delivery to public 4. Iterative processes

The Ottawa Model of Research Use

Decision-maker dilemmas impacting: evidence synthesis, knowledge translation and implementation Legislative and funding require use of high rigor evidence for programs and practice New burden on practitioners to acquire, interpret, and apply evidence Beyond skill and resources Evidence pool is often underdeveloped Additional obstacles to evidence incorporation, chief among them: Time policy time does not coincide with research time Resources funding available qualified labor pool values Culture clinical, organizational, experiential, consensus on values that impact the reception of evidence Common fallacies in evidence incorporation Locality effect “findings weren’t generate in my district with my people” Experience “we know it works”

Overcoming decision-maker dilemmas Knowledge translation is designed to reduce the burden to decision-makers But it is only as good as the preceding steps… While it is recognized that each step from question formation through dissemination is integral to knowledge translation, we have less progress in the earlier phases In a word: Early phases of the knowledge generation and synthesis process must always consider: a. the desired dissemination, translation and implementation goals and b. decision-maker obstacles

Evidence synthesis and knowledge translation are inseparable Question Refinement Synthesis Socio-political context Decision Translation and Implementation

Remainder…. How do we adapt earlier phases so that knowledge translation reaches the right people, at the right time, with the right message. ? For Cochrane that means can and how can we adapt Cochrane Review methodologies to increase the likelihood of successful dissemination? In a word: is there is a viable alternative to the gold standard systematic review

The systematic review dilemma 1. The methodologies that 3. This leaves knowledge translation and decisionmake systematic reviews makers with unattractive authoritative take time options to proceed …in years … not months, weeks, days or hours 2. Decision-makers often have trouble with evidence importation …global evidence viewed skeptically …preference for local samples, data, evaluations …. without high rigor information, …with a non-representative sample of high rigor evidence, …. with low rigor evidence.

Cochrane’s Response Adapts rapid review methods Relies on KTA’s work on rapid evidence reviews Creates new protocol that defines processes from question development to implementation

IDEALLY The rapid review is a mixed-method, interdisciplinary examination of evidence that directly addresses the “pull” needs of decision makers. Findings of rapid reviews provide a compass to understand current practice and to set the course for outcome improvements in a specific environment. Rapid reviews are tailored, fast, and implicitly designed to address implementation decisions. Error, biases, and wasted time are threats to the value of a rapid review. INCORPORATIONS TO IMPROVE EFFICIENCIES and IMPACT: Knowledge management frameworks Diagnosing true knowledge gaps Prioritization criteria Mixed methods and fields Survey methods research and cognitive interviewing Decision making Understanding of risk and probability Simulation research Implementation and evaluation research Public health and systems research Business Lean knowledge work Teaming Work place 360°evaluations

Modifying KTA to Cochrane Culture Through our 2 -year CIHR KTA research grant, we began to dive into the realm of ‘rapid reviews. ’

Local Partnership – KTA § Champlain LHIN needed assistance in developing knowledge capacity § Wanted energies directed to “push” activities; primarily knowledge intelligence services as they required expedient evidence-based answers to help direct policy, implementation, and practice decisions § “Evidence Summaries” – a form of rapid review – was developed and was iteratively refined § A series of evidence summaries (n=19) were produced (~4 -6 wks each) in response to clinical and health services questions developed with LHIN managers and stakeholders

Scoping Exercise of Rapid Reviews (Ganann et al 2010) Findings from 70 studies that authors looked at: • RRs employ a variety of methodologies; • Variety of different labels (rapid reviews, HTAs; evidence summaries; ultra rapid reviews) • Seem to vary in depth of description of methods used to abbreviate the process • Few discussed limitations (what was lost) or what bias was potentially introduced by using RR methods

No universally accepted definition of rapid review • Closest we’ve come to a definition: – Rapid review (RR) is a type of literature review produced using accelerated and streamlined systematic review (SR) methods • Using various means to short circuit the system

KTA Project with the Champlain LHIN

Rapid Review Methods § Essentially developed an abbreviated & accelerated version of current SR methods § Borrowing from Cochrane but having to make certain concessions compared to a traditional SR to accommodate expedited turnaround time (4 -6 weeks) § Out of which emerged an 8 -step rapid review process developed iteratively across development of 19 ‘Evidence Summaries’ within a 24 -month span

Rapid reviews: From start to finish it was an 8 -stage approach… 1. 2. 3. 4. 5. 6. Needs assessment (with stakeholders) Question development and refinement (PICO/TS) Protocol development and approval Systematic literature search (with limits) Screening and selection of studies (with limits) Narrative synthesis of included studies (no metaanalysis) 7. Report production phase 8. Ongoing follow-up with end users Objective: high rigour, transparency, and usability

Rapid Reviews vs. Traditional Systematic Review General Differences General Similarities - Condensed timeline - Limits set on inclusion criteria of the search including how grey literature is dealt with - Post-hoc rethinking of eligibility criteria depending on volume and applicability of evidence - Heavy reliance on systematic review evidence vs. Inclusion of primary studies (e. g. , RCTs or observational studies) Retains core values of being transparent to facilitate replication Adoption of standards to assess quality of included studies Share preference for highest quality studies

Overview of our reports: • Conducted a total of 19 rapid evidence summaries to date • 13 – focused on clinical initiatives – 9 across the field of obstetrics/gynaecology • Better Outcomes Registry & Network Ontario (BORN Ontario) – an initiative to increase capacity to plan and deliver improved maternal-child services – 6 – focused on health systems/ health services initiatives • Effectiveness of Antimicrobial Stewardship Programs in Hospitals – Requests came from various stakeholders (n=9)

Snapshot of Rapid Review report

KTA Interviews – Knowledge Users’ Impressions of Rapid Reviews Approach Stakeholders (CHAMPLAIN LHIN, BORN ONTARIO, OTTAWA HOSPITAL RESEARCH INSTITUTE (OHRI), THE OTTAWA HOSPITAL (TOH)) • Scientific Director of Performance Measurement – TOH • Chief Information Officer, Champlain LHIN • Former CEO, Champlain LHIN • Clinician, transport nurse, educator, and lactation consultant, BORN ONTARIO • Scientific Manager, BORN ONTARIO • Division Chief, Endocrinology and Metabolism, TOH • Clinical Investigator (MD), Clinical Epidemiology, OHRI, TOH • Senior Health Planning & Engagement Specialist at Champlain LHIN

Knowledge Users’ Feedback Utility • Structure/ format of summaries is important – eyes being drawn to key messages (capture reader’s attention) • Thought to be incredibly useful to have evidence at your fingertips and summarized as it was • Format was user-friendly for non-researcher decisions makers and clinicians • Format increased level of comfort with data for non-clinicians, and validated that decisions are made based in part on the evidence provided

Knowledge Users’ Feedback-2 • Notion that rapid reviews assists with having wide group of stakeholders from various backgrounds to get on the same page in terms of understanding what the latest evidence is and what it means (levels out playing field in decision-making so process less prone to group think scenarios) • Directly influenced practice change in local delivery of care (Example: summaries are being used to move forward on the regional plan on delivering diabetes care)

Knowledge Users’ Feedback-3 • Approach has helped build relationships between those creating the summaries and the implementers - key ingredient to ensure ongoing dialogue • Notion that the evidence is coming from an “objective, third party” lens – and not from one particular angle/clinical slant, made the findings of the rapid reviews seem more credible • Rapid reviews were used to help set the context for LHIN discussions, and were key to planning • Also, used to lay the groundwork as part of a broader environmental scans; ultimately helping with prioritizing of LHIN projects

Knowledge Users’ Feedback-4 Barriers • Policy and practice implementation difficult – taking general evidence and applying it locally found to be challenging • System issues provide obstacles (for example, such as sites where they only have OR times for 1 -day/ week which then controls when surgeries can happen –even though that might not match with when it should be done) • Unfavourable attitudes towards evidence requires behaviour change efforts (interventions difficult to implement) • Taking the scientific evidence and translating it into a userfriendly format for a variety of end-users (dissemination science applied)

KTA Dissemination • Reviews are disseminated on our website (www. ohri. ca/kta), to end-users directly, and when possible, through publication

KTA Publications of SRs J n i article ry 2012; d e t i c hly brua g i e h F t n s i Mo ads lished o l b n u w p o e sinc iews/d v 7 3 2 5

KTA Teaching & Training • To date, we have delivered several workshops

Local Program Development The Ottawa Hospital Technology Assessment Program (TOHTAP) • Hospital-based knowledge support service with focus on rapid reviews • Knowledge synthesis + local data + economical evaluation • Inaugural year with possible longterm funding commitment (OHRI, TOH Performance Management & Patient Safety groups)

Cochrane Rapid Response. .

International Program Development Cochrane Innovations • Newly established program by Cochrane Collaboration with primary goal of providing a mechanism by which Collaboration can respond to requests for commissioned reviews

Cochrane Response • Rapid reviews are one type of commissioned review, for which the Cochrane version is referred to as “Cochrane Response” • Currently, small global consortium has been assembled to develop materials to assist in the conduct of Cochrane Responses – to date relied heavily on our KTA approach • Authors will be experienced Cochrane authors and active members of the Cochrane Response Consortium

Cochrane Response • Each requires at least two reviewers, one information retrieval specialist, content expertise, and the necessary input of the client • Process adds in 24 -hour peer review periods with access to input from across the many content and methods experts that make up the collaboration (Cochrane’s 55 Review Groups, 14 Fields, and 12 Methods Groups) • Each Response is placed on an 8 -week schedule and has intermediate deliverables scheduled throughout this period • There is an intention to publish these rapid reviews through the Cochrane Library, making them available for other decisionmakers

Cochrane Response Process 11 -stage process developed based on Cochrane principles, and a review timeline of 8 -weeks 1. Topic Refinement 2. Contracts 3. Problem Statement 4. Protocol Development 5. Literature Search 6. Screening & Study Selection 7. Data Mining & Extraction 8. Synthesis 9. Report Contruction 10. Contingency Phase 11. Additional Products or Services

In Summary IDEALLY The rapid review is a mixed-method, interdisciplinary examination of evidence that directly addresses the “pull” needs of decision makers. Findings of rapid reviews provide a compass to understand current practice and to set the course for outcome improvements in a specific environment. Rapid reviews are tailored, fast, and implicitly designed to address implementation decisions. Error, biases, and wasted time are threats to the value of a rapid review. INCORPORATIONS TO IMPROVE EFFICIENCIES and IMPACT: Knowledge management frameworks Diagnosing true knowledge gaps Prioritization criteria Mixed methods and fields Survey methods research and cognitive interviewing Decision making Understanding of risk and probability Simulation research Implementation and evaluation research Public health and systems research Business Lean knowledge work Teaming Work place 360°evaluations

Evidence and decisions: variables at play Question Type of decision: • Funding • Service • Refinement or change of practice Consensus on utility of question Refinement Access and Management of stakeholder groups Processes used to refine PICO/ST Synthesis Resources for evidence synthesis: • Qualified labor • Training integrity • time • funding • Generalizability • Local context Diversity of Stakeholder group Decision Distance between evidence and current practice Competing priorities: Introduce funding Do nothing Translation activities windows missed Decision maker characteristics: Goals, Position, Agency, Beliefs, Experiences Socio-political context Translation and Implementation

Cochrane Collaboration Ottawa Hospital Research Institute: u. Ottawa EPC Knowledge to Action Canada AHRQ Effective care program EPCs, DEc. IDE Cochrane Innovations US CONGRESS George Mason University Cochrane Policy College US Representative: VA 11 th District Connelly Core EDD team Dept of Medicine I INOVA Health Systems NOVA HEALTH SYSTEMS Chief Medical Officer CMO’s hospitals and Care Centers Congressman Moran VA 8 th District Targeted patient population within Northern Virginia’s 11 th District Congressman Wolf VA 10 th District Extensions of sustainable relationship into Virginia’s 8 th and 10 th Districts

Rigor requirements needed for all aspects of knowledge generation, rapid review, and dissemination Empirical assessment of outcome improvement Do those receiving intervention perform better Retain knowledge Change practice Share information Are practices influenced Are systems refined Is money saved How do moderators at each stage affect reception of knowledge What happens when you manipulate conditions Assert funding unto scheme Introduce alternative therapy Skip one or more of the steps What is the best format of findings display (see Dr. Probability next) Encourage use of limited resources for high utility questions Does partnership and ongoing delivery encourage change over inserting external evidence? Decision making training Do decision makers understand evidence? Are they able to use it in light of competing priorities? Can they use evidence to ensure greatest clinical, fiscal, social utility? Did change happen? Are outcomes better Are decisions Next steps: parsing the variables and assessing their impact so that knowledge translation is truly calibrated for: Type of decision Type of consumer Current contextual factors (local vs. global) Manipulate parameters Conduct random assignment to knowledge products Conduct implementation and outcome evaluation Mixed methods

But what about the medicalisation of 1000 people?