Critical appraisal in systematic reviews and rapid reviews

Critical appraisal in systematic reviews and rapid reviews For environmental policy and practice Geoff Frampton Southampton Health Technology Assessments Centre University of Southampton, UK

“A rapid review is a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting a variety of methods to produce evidence in a resource-efficient manner” → Systematic review methods are the benchmark for rapid review methods

Part 1: Critical appraisal in systematic reviews

Random sample of 50 environmental policy and practice systematic reviews identified from Collaboration for Environmental Evidence CEEDER database 21 February 2021

Random sample of 50 environmental policy and practice systematic reviews identified from Collaboration for Environmental Evidence CEEDER database 21 February 2021 SR publication year Proportion of SR with no critical appraisal stage Proportion of SR with no PRISMA or ROSES checklist 2018 (n=20) 60% (12/20) 90% (18/20) 2019 (n=17) 71% (12/17) 76% (13/17) 2020 (n=13) 69% 92% (12/13) (9/13) Overall, 77% (of 43) non-CEE SRs and 0% (of 7) CEE SRs had no critical appraisal stage

Among the 17 systematic reviews that conducted critical appraisal… Critical appraisal aspect Total SRs CEE SRs Focused on internal validity (good) 59% (10/17) 71% (5/7) Included all key internal validity domains (good) 47% (8/17) 29% (2/7) Conflated “quality” constructs (bad) 41% (7/17) 29% (2/7) Used numeric summary scores (bad) 29% (5/17) 0% Did not fully inform the data synthesis (bad) 71% (12/17) 86% (6/7) Unclear criteria or unexplained judgements (bad) 88% (15/17) 86% (6/7) (N=17) (N=7)

What is the “quality” of a quantitative study? Selection bias Precision (random error) Internal validity (systematic error) Whether peer reviewed External validity Analytical accuracy Detection bias Attrition bias Clarity of reporting Appropriateness of question Other types of confounding

Steps to improve critical appraisal guidance – work in progress

Steps to improve critical appraisal guidance – work in progress ● Plan – Conduct – Report – Apply framework ● Based on overarching core principles that critical appraisal should be: Focused, Extensive, Applied and Transparent (FEAT) ● Consistent with existing fit-for-purpose tools e. g. Cochrane Ro. B, ROBINS-I, etc

Steps to improve critical appraisal guidance – work in progress ● Plan – Conduct – Report – Apply framework ● Based on overarching core principles that critical appraisal should be: Focused, Extensive, Applied and Transparent (FEAT) Focused – Critical appraisal should be focused on a relevant construct. Logically for quantitative reviews this should be systematic error (internal validity)

Steps to improve critical appraisal guidance – work in progress ● Plan – Conduct – Report – Apply framework ● Based on overarching core principles that critical appraisal should be: Focused, Extensive, Applied and Transparent (FEAT) Extensive – All relevant domains of the construct should be captured (e. g. all relevant domains of bias and confounding if the focus is on internal validity)

Steps to improve critical appraisal guidance – work in progress ● Plan – Conduct – Report – Apply framework ● Based on overarching core principles that critical appraisal should be: Focused, Extensive, Applied and Transparent (FEAT) Applied – The critical appraisal output should inform the data synthesis in an appropriate way, not conflating the assessment constructs

Steps to improve critical appraisal guidance – work in progress ● Plan – Conduct – Report – Apply framework ● Based on overarching core principles that critical appraisal should be: Focused, Extensive, Applied and Transparent (FEAT) Transparent – The definition of the assessment construct and rationale for judgements made for each study and outcome should be reported

How does critical appraisal in environmental policy and practice systematic reviews fare against the FEAT principles? For the 17 systematic reviews in the random sample which reported critical appraisal Systematic reviews (n=17) FOCUSED EXTENSIVE on internal capturing all validity relevant bias domains APPLIED to inform data synthesis appropriately TRANSPARENT reporting of definitions and judgements Non-CEE (n=10) 40% (n=4) 20% (n=2) 0% CEE (n=7) 71% (n=5) 14% (n=1) 29% (n=2) 14% (n=1)

Part 2: Critical appraisal in rapid reviews

4/13 rapid reviews conducted critical ← appraisal

How do rapid review guidelines fare against the FEAT principles?

How do rapid review guidelines fare against the FEAT principles? Transparency → Fail “…robustness (quality)…” External validity? ? ? Internal validity (+ other constructs)? ? ?

How do rapid review guidelines fare against the FEAT principles? Focus, Extent, Application, Transparency → Fail

FEAT principles could assist planning and critiquing of critical appraisals for any type of evidence review and topic Focus ● Determine & justify the assessment construct (if any) to be used ● Internal validity is a logical focus for effectiveness reviews Extent ● Justify which domains of the construct are to be assessed Application ● Ensure that different assessment constructs are not conflated ● Ensure that the critical appraisal informs the data synthesis ● Use a meaningful output, not an undefined “quality” scale Transparency ● Provide a rationale for judgements made

Conclusions Need to understand reasons for poor uptake of critical appraisal in systematic reviews Complying with the FEAT principles could help to make rapid reviews more reliable than some current systematic reviews But risk of repeating the same mistakes if guidelines on rapid reviews are not considered carefully and evidence -based Acknowledgements: Paul Whaley, Lancaster University for valuable discussions on FEAT principles

Additional/spare slides

Environmental policy and practice systematic reviews N=50 randomly selected from CEEDER database 21 Feb 2021 Of the 43 non-CEE systematic reviews 15 assessed publication bias: Assessed both critical appraisal and publication bias: n=5 Assessed critical appraisal only: n=5 Assessed publication bias only: n=10

Used the Koricheva & Gurevitch (2014) checklist for meta-analyses – which covers publication bias but not critical appraisal… so this QA of meta-analyses entirely misses critical appraisal but emphasises PB

Problems with combining constructs and using numerical scores ● Not transparent which construct(s) are assessed in an overall quality score ● Constructs are nominal categories unsuitable for mathematical operations ● Different constructs and domains within a construct cannot be assumed to have the same weight (NB Jadad score for randomised controlled trials) ● Combining constructs may dilute the representation of systematic error ● Scores are not generalisable beyond a specific critical appraisal ● Information on key limitations of studies is lost → research waste

Why such poor uptake of the critical appraisal stage in environmental policy and practice systematic reviews? ● Range of study designs which have different validity issues, but no agreed classification or inventory of designs ● Many tools and checklists exist but mostly developed in health research – are they applicable? which to choose? ● Published SRs vary widely in the approaches used – what is good practice? ● Inconsistency in terminology, e. g. validity versus quality ● Lack of detailed guidelines for the environmental policy and practice area ● Inadequate knowledge / editorial standards / peer review