Implementation Research Methods A Primer EECA INTERACT November

Implementation Research Methods: A Primer EECA INTERACT, November 19, 2019 Stefan Baral, MD MPH FRCPC Department of Epidemiology Johns Hopkins School of Public Health

Disclosures n I have no financial disclosures 2

Overview n n n Definitions Characteristics Conceptual Frameworks Evidence and Outcomes Methods Qualitative Quantitative Descriptive Analytic Experimental Conclusions 3

A. Implementation Research Principles The Scientific Method

Definitions n Implementation Science in HIV Implementation science is a multi-disciplinary field that seeks generalizable knowledge about the behaviour of stakeholders, organizations, communities, and individuals in order to understand the magnitude, reasons for and strategies to close the gap between evidence and routine practice for health in real world contexts Key Themes u u u Multidisciplinary Generalizable Multiple stakeholders Closing gap between evidence and practice Real world contexts Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013; Odeny, Padian, Doherty, Baral, Beyrer, Ford, Geng, Definitions of implementation science used in the HIV/AIDS literature: a synthetic review. The Lancet Infectious Diseases, In Press, 2015 5

Implementation Research and Other PH Study Designs Source: Olakunle Alonge, Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013; https: //catalyst. harvard. edu/pathfinder/t 2 detail. html 6

Characteristics of Implementation Research n Findings are Warranted to Inform Policy/Program There is “sufficient evidence” to support the conclusions of the work u What is sufficient evidence? n Transparency of Methods Support Critical Assessment of the Study u Whether processes are adequate u Conclusions justified u Repeatability Don’t be afraid of “failure” u A well done study is still a success in terms of generating generalizable knowledge 7

Traditional Scientific Method http: //www. sciencebuddies. org/science-fair-projects/project_scientific_method. shtml 8

Differences with IR Competencies on a IR team: Research Methodologist u Qual, Quant, Mixed Methods Ministry, Government, Agencies u Either as members of team or study oversight committee Health Professionals u Involvement of health professionals from study settings Communications Public Health Professionals u Health Commissioner/Associate Health Commissioner u Public Health Inspector/Public Health Nurse Privacy Expert Stakeholder Assessment u Community http: //www. who. int/tdr/publications/year/2014/ir-toolkit-manual/en/ 9

Traditional IR Approaches Active Consultation Appropriate Team, Active Consultation Active Consultation http: //www. sciencebuddies. org/science-fair-projects/project_scientific_method. shtml 10

B. Evidence in Implementation Research Conceptual Frameworks Tension between External and Internal Validity Implementation Research Reporting

Conceptual Frameworks Commonly used in IR n RE-AIM Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance n Stages of implementation National Implementation Research Network Exploration and Adoption, Program Installation (Prep), Initial Implementation (pilot/adapt), Full Implementation (>50% coverage), Sustainability n Consolidated Framework for Implementation Research Intervention Characteristics, Inner Setting, Outer setting, Individuals in the Intervention, Implementation process Many others…. 12 n Source: Glasgow et al 1999, National Implementation Research Network, 2005, Damschroder, 2009

Outcomes in Implementation Research Implementation Outcomes Acceptability Adoption Appropriateness Costs Feasibility Fidelity Penetration Sustainability Services Outcomes Efficiency Coverage Equity Responsiveness Health Outcomes Clients Outcome Satisfaction Symptomatology Function Population. Based Incidence of diseases Morbidity Mortality DALYs Source: Olakunle Alonge, Proctor et al 2011 13

Implementation Outcomes Implementation Outcome Acceptability Working Definition* Related terms** Perception among stakeholders that an intervention is agreeable Related factors: (e. g. Comfort, Relative advantage, Credibility) Adoption Intention, initial decision, or action to try to employ a new intervention Perceived fit or relevance of the intervention in a particular setting or for a particular target audience (e. g. provider or consumer) or issue The extent to which an intervention can be carried out in a particular setting or organization The degree to which an intervention was implemented as it was designed in an original protocol, plan, or policy Uptake, Utilization, Intention to try, Appropriateness Feasibility Fidelity Implementation cost Coverage Sustainability Incremental cost of the implementation strategy Degree to which the population that is eligible to benefit from an intervention actually receives it. The extent to which an intervention is maintained or institutionalized in a given setting. Source: Proctor et al 2011; Peters, Adams, Alonge et al 2013 Relevance, Perceived fit, Compatibility, Perceived usefulness or suitability Practicality, Actual fit, Utility, Trialability Adherence, Delivery as intended, Integrity, Quality of programme delivery, Intensity or dosage of delivery Marginal cost, Total cost*** Reach, Access, Service Spread or Effective Coverage, Penetration Maintenance, Continuation, Routinization Institutionalization, Incorporation 14

Evidence Supporting Interventions n Donabedian Approach Process u The Traditional Gold Standard for M&E u Is system efficient? Counting the actual products distributed, people trained, etc l Condoms, Peer Educators, Paralegals, etc. Structure u Structural Outcomes of the Intervention Health Systems, Health Policies, etc Outcome u Emerging Gold Standard… u What is happening with outcome of interest? Impact! l Efficacy vs Effectiveness 15

What is Sufficient Evidence? n Evidence-based medicine is a global standard Double-Blinded (DB) RCT is gold standard n Evidence-based PH interventions should also be a global standard Often limited evidence, PH decision still needs to be made Precautionary Principle for PH? u When there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost -effective measures to prevent environmental degradation 16

Tension Between Internal and External Validity n Internal Validity Minimal study biases suggesting confidence in ultimate conclusion of the study u Efficacy Research n External Validity Generalizability of ultimate findings to broader population and contexts u Implementation Research n To develop guidelines Need to characterize u Efficacious u Effective u Sustainable and Scalable programs

Tension in Research about Validity n Traditional Approach is to establish internal validity with certain study designs and then have studies focused on external validity Internal Validity u Phase 1 (Safety), Phase 2 a/b (tolerability, TOC), Phase 3 (Efficacy) External Validity u Phase 4 (Post-Marketing)

Evidence Supporting Implementation n Traditional Question for Clinicians/Programmers Does it work? What is effect size? u Should I use it? n Implementation Questions How, when, why, and where does it work? What factors influence effectiveness? u Should I use it? How should I use it? 19

New Models of Research? n Efficacy Research (and guideline development) generally happens prior to implementation research Are there more time-effective approaches to integrate implementation research with efficacy research u Assess barriers/facilitators to intervention uptake acceptance/adoption/routinization u Diagnose quality gaps Fidelity u Characterize Sustainability Maintenance, Cost-Effectiveness

Real World Example Global Health Editorial. Women’s Groups to Improve Maternal and Child Health Outcomes: Different Evidence Paradigms Toward Impact 21 at Scale. 2015

The “Four P’s” of Implementation Research Context Internal validity intervention design and specification Pedantic Purile Pragmatic Populist Generalizability Modified from Elvin Geng, Source: David Aron Counterfactuals

Reporting of Implementation Evidence n n n Background Care or quality gap that the intervention seeks to address and evidence supporting the intervention Theory behind the intervention and how theory influenced the study design Setting Describe the study setting Describe any changes or modifications required to adjust to the intervention Report readiness to change (also those not eager to change) How did the setting enable or hinder the implementation? Participants Methods of Recruitment (to clinical service/program) Eligibility Criteria (ie who is the target population) 23

Reporting on Implementation n Intervention Frequency, duration and intensity of the intervention Mode of delivery and materials used What is the relationship of components to theory Provide detailed characteristics of those promoting and undertaking the intervention What were the processes for ensuring intervention fidelity? Describe the intervention received by control group (not just describe as ‘usual care’) Describe Costs Pinnock, Developing standards for reporting implementation studies of complex interventions (Sta. RI), BMC, 2015 24

Implementation Reporting in the Context of HIV/AIDS treatment Hickey, , Odeny, Petersen, Neilands, , Baral, Geng. Specification of Implementation Interventions to Address the Cascade of HIV Care and Treatment in Resource Limited Settings: a Systematic Review. Under Review 25

C. Implementation Research Methods Qualitative Experimental Designs

Qualitative Implementation Research n Instrument fidelity – Measure the Right Things n Treatment integrity – Fidelity of Implementation Mixed Methods in order to assess the fidelity of interventions, treatments, or programs n Describe Contexts or Settings surrounding Implementation n Understand the meaning behind Implementation Explore the “why” or “how” an intervention works

Observational Designs n n n Experimental designs are not always Feasible Affordable Ethical making Unanticipated events create quasi-experimental conditions individuals experiencing in certain places from one period to the next receive different interventions not based on their characteristics, but based on timing/chance Hierarchy to observational evidence (inferior superior) 1. 2. 3. 4. One exposure group, no control Two groups (including one control), no pre-test, post-test only Two groups (including one control), pre-test / post-test Interrupted time-series design 28

Observational data – analytic approaches n Changes over time Post-pre u n Difference in differences n least rigor post-pre comparison in exposed compared to nonequivalent control group Approaches building on differences within or between individuals over a small period of time based on rapid exposure changes (e. g. new policy) or random differences in exposures assigned Regression discontinuity Interrupted time series Instrumental variable 29

Experimental Studies n n n Explanatory (Traditional Gold Standard) Understand explain benefit of new model of implementation of intervention under controlled conditions Maximize internal validity Pragmatic Trials Focus on the intervention in routine practice Intentional maximization of variability in how study is implemented u Variability of research settings (communities, practice settings, types of providers, patients) Maximize external validity Adaptive Designs Emerging area of study that attempt to balance internal and external validity

Pragmatic Trials n n n Testing a new intervention while maximizing external validity Formative Period Qualitative work, some descriptive or analytic observational work Consider different types of outcomes of effectiveness Directly Measured u Health Outcomes u Service Outcomes u Implementation Outcomes u Resources Institutional Human Financial l Cost-effectiveness, Cost-Utility, Cost. Minimization, Cost-Benefit, etc Indirect Assessment/Modeled Benefits u Increasing use of Mathematical Models to Scale Results for potential longer term outcomes, etc.

PROUD Pr. EP RCT n n n A multi-centre, open label randomized design to immediate or deferred inclusion of pre-exposure prophylaxis as part of the package of HIV risk reduction interventions. Setting: Genitourinary Medicine Clinics (Ie STI Clinics) Intervention: HIV testing, Dx/Tx of STI, Condoms, PEP +/- Truvada Pr. EP Main Outcome Time to Accrual of 500 Participants Retention at 12/24 Months Secondary Outcome HIV Incidence over a 12 month period (powered on this) Renal function, HIV resistance among men who acquire HIV, Adherence, Risk Compensation, Facilitators/Barriers to Adherence http: //www. proud. mrc. ac. uk/pdf/PROUD_protocol%20 v 1%203%2014 October 2014. pdf

PROUD Individual RCT http: //www. proud. mrc. ac. uk/pdf/PROUD_protocol%20 v 1%203%2014 October 2014. pdf

Effectiveness-Implementation Hybrid Trials n Goal Measure markers of implementation and impact in the same study n Three Broad Designs Differentiated by prioritization of data collection u Type 1 1 st priority - Impact of health intervention 2 nd Priority - Measures of implementation l Feasibility/Acceptability using qualitative/mixed methods u Type 2 Equal priority to impact and implementation u Type 3 1 st priority – Implementation l Fidelity of intervention, measures of adoption, etc. 2 nd Priority Impact of Health Intervention

Stepped Wedge Cluster Randomized Designs n Method Assess baseline situation in all communities, but randomly phase in intervention activities in steps, evaluating impact of intervention time on outcomes http: //www. biomedcentral. com/1471 -2288/6/54; http: //www. trialsjournal. com/series/Stepped. Wedge

Stepped Wedge Designs n Pros Differences in exposure time allow each community site to receive the intervention n Mixed n n Ethics of stepped wedge Designs n Some believe more ethical to give intervention to all and more feasible to implement, others believe trial not warranted if success of intervention is certain and standard of care can be justified so why not assess more cleanly with parallel design Cons Analysis concerns around when an intervention starts – e. g. if a community starts receiving the intervention today but takes 2 months to scale up and reach a substantial number of people, exposure time will be diluted as everyone starts receiving the exposure at the same time within the cluster If the outcome cannot be expected to happen over the time period of one step, stepped wedge designs will be underpowered

Adaptive Designs For Implementation Studies n n Adaptive Intervention/Adaptive Implementation Strategy Specific decision rules for the implementation of an interventions based on individual/community needs Trying to maximize both internal and external validity Trial Design Sequential, multiple assignment, randomized trials (SMART) u Use outcome data to inform the implementation of the intervention being evaluated Can be at multiple steps Adaptive does not mean Flexible http: //methodology. psu. edu/ra/adap-inter

SMART Study Example Responde rs DT P NVS FSW ICM Nonresponders Responde rs Nonresponders NVS – Non virally suppressed So. C – Standard of Care DTP – Decentralized ART Program ICM – Individualized Case Management Dark red box – Randomization So. C DT P ICM+ DTP So. C ICM ICM+ DTP

Power and Sample Size Calculations n P&S Calculations for IS studies are complicated Powered to assess the “preponderance of evidence” of the benefit of interventions u Most realistic, but murky Powered for at least primary outcome (Eg. Viral suppression) u Cleaner, but is this really implementation research? Powered on Outcome and Implementation Outcomes u Limited resources, etc.

De-Implementation/Ex-novation Research n n n The science of dissemination and implementation confronts two problems Getting wider uptake of evidence-based interventions in clinical or public health practice Elimination from clinical or public health practice of tests and interventions that use resources without enhancing patient outcomes As a field, we focus more on increasing interventions than we do reducing unnecessary ones More incentive to discover new tools than to try and more politically sensitive to try and remove services for folks De-Implementation Science Methods Use many of the same experimental methods (CRCT, SW, etc) but in reverse

Summary n Implementation Research Seeks Generalizable Information intending to Close Gap Between Evidence and Practice n Tension Between Internal and External Validity What is most important for you and your stakeholders n Rapid Evolution of Experimental Approaches in Implementation Research 42