RESEARCH SYNTHESIS The Core Methodology of EvidenceBased Reviews

RESEARCH SYNTHESIS: The Core Methodology of Evidence-Based Reviews Susan N. Labin, Ph. D. susan@susanlabin. com Susan N. Labin-AEA 2010

Research Synthesis: What & Why? Secondary data analysis-aggregates findings from primary research n Establishes base of existing knowledge n Informs future policies, programs, practices, & research n Susan N. Labin-AEA 2010 2

Types of Syntheses n Broad-Based: from variety of designs & types of data – GAO – CDC: Guide to Community Preventive Services n Meta-Analysis: only from randomized control trials (RCT) Susan N. Labin-AEA 2010 3

Broad-Based Research Synthesis: GAO & CDC Moderately systematic n Qualitative & quantitative data n Draw from a variety of designs n Consensus building-between academics and practitioners n Susan N. Labin-AEA 2010 4

Research Synthesis: Systematic Decision Rules n Research Synthesis – Broad-based and meta-analysis – Systematic decision rules – Six basic steps, e. g. selection & inclusion criteria n Traditional Literature Reviews – Broad-based – Less systematic decision rules Susan N. Labin-AEA 2010 5

Brief History: Designs @ Primary & Secondary n n n Evaluation 1960’s – RCT “gold standard” from physical and biological sciences 1970’s and 1980’s interest in synthesis and meta-analysis for social sciences Resolve contradictory findings Improve external validity or generalizability of small sample RCTs Meta-Analysis physical & biological roots Susan N. Labin-AEA 2010 6

Evidence-Based Practice Reviews (EBPR) As synthesis-assess evidence to inform policy & practice n Who: predominately federal n How same & different from synthesis? n How can synthesis inform EBPR? n Susan N. Labin-AEA 2010 7

EBPR Who Is Doing Them? 1. 2. 3. 4. 5. 6. Agency Healthcare Research & Quality Centers for Disease Control & Prevention 2 Office Juvenile Justice & Delinquency Prevention Substance Abuse & Mental Health Services Administration US Dept of Ed Coalition for Evidence-Based Policy (nonfederal) Susan N. Labin-AEA 2010 8

Syntheses & EBPR: Common Steps & Decision-rules n n n Many of same steps Quality ratings Inclusion-Design – – – Some restrict to only RCT Others include quasi-experimental, observational Some only RCT can get highest rating, most RCT highest ratings Susan N. Labin-AEA 2010 9

EBPR Differs from Synthesis Inclusion-some EBPR include only those with positive outcomes n Only a few EBPR aggregate analyses across studies n Less emphasis w EBPR on reliability between reviewers n Susan N. Labin-AEA 2010 10

Addressing Practice Challenges to RCT Social interventions linked with reinforcing social interactions or culture- address w group/place/cluster/saturation trials n Assumptions re replicability of protocol or intervention n Feasibility n Susan N. Labin-AEA 2010 11

RCT Challenges for Social & Behavioral Sciences Inherent differences between physical and biological sciences and social and behavioral sciences n Lack of blinding and double blinding n Relevance of individual choice n Susan N. Labin-AEA 2010 12

Comparing Results: RCT versus Quasi Designs Complex: quasi may underestimate (Shadish) n Same labor market, same questionnaire quasi & exp give “fairly close” results n Susan N. Labin-AEA 2010 13

Comparing Results: RCT vs Quasi Designs cont. n Lipsey & Wilson (2001) – > 300 M-A; > 16, 000 ind studies – Exp vs Quasi not much difference – One-group pre-post over estimates compared to quasi – Measurement =. 07 proportion of variance – Exp vs Quasi=. 04 Susan N. Labin-AEA 2010 14

How To Strengthen EBPR? n Include negative results to inform counterfactual – – n Participatory and transparent reviews – – n How consistent results What else could be contributing to the effects Quality criteria and decision rules Reliability & resolving discrepancies Common measurement-include criteria of respondent meanings Susan N. Labin-AEA 2010 15

How To Strengthen EBPR cont. n Aggregate across projects to improve – – n Generalizability > Confidence in “best practice” designation Choice of projects for replication Knowledge Base Develop & use technologies for categorizing studies and depicting patterns of results. Susan N. Labin-AEA 2010 16

References n Labin, S. 2008. Research Syntheses: Toward Broad-Based Evidence in Fundamental Issues in Evaluation, (eds. ) Nick L. Smith and Paul R. Brandon, Guilford Press, NY. GAO-30 -10, 2009. Program Evaluation: A n Websites for the Effective Practice Reviews n Variety of Methods Can Help Identify Effective Interventions Susan N. Labin-AEA 2010 17

THANK YOU Susan N. Labin, Ph. D. 301. 564. 0764 susan@susanlabin. com www. susanlabin. com Susan N. Labin-AEA 2010 18