Unit 5 EvidenceBased Practice Evidence or Orthodoxy INFORMATION

Unit 5: Evidence-Based Practice: Evidence or Orthodoxy? INFORMATION FROM DUNCAN, B. L. , MILLER, S. D. , WAMPOLD, B. E. , & HUBBLE, M. A. (EDS). (2010). WASHINGTON, DC: AMERICAN PSYCHOLOGICAL ASSOCIATION.

Evidence-Based Practice “Believe those who are seeking the truth; doubt those who find it” (p. 167). Andre’ Gide, as cited in Duncan, Miller, Wampold, and Hubble (2010).

What is Evidence-Based Practice? �EBP is the “integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences” (APA, 2006, p. 73) �Dissemination of research into practice is an often overlooked area in terms of evidence base (Ohmer & Korr, 2006) �Ideal that clinical practice will be informed by empirical research �Scientist-practitioner model �EBP focuses on efficacy and effectiveness of interventions

Evidence-Based Treatments �EBTs—lists of effective or model programs Blueprints for Violence Prevention series https: //www. ncjrs. gov/pdffiles 1/ojjdp/204274. pdf National Registry of Evidence-Based Programs and Practices http: //www. nrepp. samhsa. gov/ APA Society of Clinical Psychology http: //www. apa. org/about/division/div 12. aspx

The “Know-Do” Gap �Limited use of EBTs Faulty graduate education Resistance among clinicians / community agencies Ineffective dissemination practices �Opposition to EBT Reflects political / theoretical bias rather than effective treatment Pressure to limit interventions to EBTs erodes professional autonomy Lack of trust in researchers (i. e. drug companies)

“If we fund it…” �Translational research – process by which scientific discovery is successfully institutionalized Dissemination, implementation, and diffusion �Organizational and financial incentives to implement EBTs State funding to support EBTs State of Washington: assigned grades to services with “proven benefits” to clients; grades affect coverage decisions Mandates for inclusion of EBTs in mental health and substance abuse practice

EBP and EBT Evidence Based Practice Evidence Based Treatment � Model of practice � Programs that meet � Reflects understanding certain evidentiary criteria � Depicts confidence in available evidence that scientific evidence is tentative � Appeals to those who value clinicians’ autonomy and individualized treatment decisions � Appeals to those who believe more structure and consistency is needed in mental health settings

The Achilles Heel �Proliferation of unscientific research syntheses �Both EBP and EBT rely on unscientific syntheses of evidence about what works �Information on EBP and EBT incomplete and potentially misleading

Methods of Research Synthesis

Research Synthesis �Cannot rely on any single study to provide definitive information about treatment effects �Outcome studies: Vary in methodology Vary in credibility May produce conflicting evidence �Synthesis of results across studies: Places findings in context Provides opportunity to investigate reasons for variations in outcomes across treatments, samples, measures and settings

Traditional, Narrative Reviews �Begins with question about available evidence on a topic Identify pertinent published studies through databases Read material and write narrative summary of studies / results Draw conclusions about similarities / differences among studies �Limitations of Narrative Reviews: Based on unspecified samples not representative of all credible sources Rely on convenience samples that may be biased toward positive results Narrative reviews rarely explain how studies were selected Reviewers rarely explain how conclusions were drawn about overall trends

Bias in Research Synthesis � Three sources of bias: Arise in original studies Design and implementation problems lead to overestimation / underestimation of effects (threatens validity) � Allegiance effects when interventions studied by advocates � Arise in reporting and dissemination of results Investigators more likely to report positive / statistically significant results � Studies with positive results more likely to be accepted for publication than those with null / negative results � Arise in review process itself Peer reviewers may be biased against manuscripts that counter their expectations / theoretical perspectives � Studies with positive results published more rapidly and cited more often � � Selective reporting, publication, dissemination, and citation of positive results make them more visible and available than other equally valid findings

Systematic Reviews �Process of utilizing measures to control biases in research synthesis Transparent Intentions and Methods � Protocol for review developed in advance � Steps and decisions carefully documents � Conflicts of interest and sponsorship arrangements disclosed Explicit Inclusion and Exclusion Criteria � Specify in advance study designs, populations, interventions, comparisons, and outcome measures included / excluded � Documentation of reasons for exclusion � Allows for replication or extension of review

Systematic Reviews Search Strategies � Identify electronic databases and develop keyword strings � Effort to locate gray literature (unpublished / hard to find studies) � Search process and results documented carefully Interrater Agreement on All Key Decisions � Two or more raters review all citations and abstracts � At least two independent raters decide on full-text retrieval, study inclusion / exclusion, and coding � Raters compare notes, document differences, document decisions Systematic Extraction of Data From Original Studies � Data extracted onto paper / electronic coding forms � Analysis of data / synthesis of results � Data forms bridge between primary research and research synthesis

Analysis of Study Qualities � Aspects of methodology related to validity of study assessed individually � Use of separate assessment of different study qualities � Campbell’s threats-to-validity approach Analysis of Study Results � Findings represented as effect sizes if possible Effect size - group of statistics that express strength and direction of an effect or relationship between variables � Ex: standardized mean difference, odds ratio, risk ratio, correlation coefficient � Documentation of data and formulas used for effect size calculations

Synthesis of Results � Combine results across studies through transparent methods Meta-analysis Reporting of Results � QUOROM statement – developed to improve reports on systematic reviews and meta-analyses Includes checklist of items to be reported Flow diagram for authors to use to describe how studies identified, screened, and selected Updating � Systematic reviews should be updated regularly Cochrane Collaboration (http: //www. cochrane. org) Campbell Collaboration (http: //www. campbellcollaboration. org)

Current Practice “SCIENCE IS SUPPOSED TO BE CUMULATIVE, BUT SCIENTISTS ONLY RARELY CUMULATE EVIDENCE SCIENTIFICALLY” (CHALMERS ET AL. , 2002, AS CITED IN DUNCAN ET AL. , 2010)

�Practice of research synthesis is not well connected to science of research synthesis Most reviews not informed by research on advantages / disadvantages of various approaches to identifying, analyzing, and synthesizing empirical data Systematic reviews / meta-analysis are becoming more common Traditional reviews still prevail in social sciences

References American Psychological Association (2006). Evidence-based practice in psychology. American Psychologist, 61, 271 -285. doi: 10. 1037/0003 -066 X. 61. 4. 271 Duncan, B. L. , Miller, S. D. , Wampold, B. E. , & Hubble, M. A. (Eds). (2010). The heart & soul of change: Delivering what works in therapy (2 nd ed. ). Washington, DC: American Psychological Association. Ohmer, M. L. , & Korr, W. S. (2006). The effectiveness of community practice interventions: A review of the literature. Research on Social Work Practice, 16, 132145. doi: 10. 1177/1049731505282204