A Review of Methods used to Quantify Effect

A Review of Methods used to Quantify Effect Sizes in Clinical Trials Joanne Rothwell University of Sheffield PSI Conference, London, May 2017 1

Outline 1. Aims of Research 2. Methods 3. Results 4. Limitations 5. Further Work PSI Conference, London, May 2017 2

Aims of Research To investigate: • which methods are commonly used and reported for eliciting the target effect size • how “accurate” the target effect size is compared to the observed effect size • whether particular clinical areas are consistently over- or under-achieving the target effect size • examples of well-justified sample size calculations and target effect size elicitation. To be completed. PSI Conference, London, May 2017 3

Aims of Research Standard sample size formula for superiority trials Fixed Most sensitive variable Estimated PSI Conference, London, May 2017 4

Aims of Research Standard sample size formula for superiority trials Fixed If d is halved, the sample size quadruples Estimated PSI Conference, London, May 2017 5

Methods • Why is this research useful? PSI Conference, London, May 2017 6

Methods • How was this research completed? – Review the Health Technology Assessment (HTA) reports for randomised controlled trials – Extract data from the reports over the period 2006 -2016 PSI Conference, London, May 2017 7

Inclusion/Exclusion Criteria for Reports PSI Conference, London, May 2017 8

Report Selection 994 reports 310 reports excluded: � Published before 2006 684 reports 507 reports excluded: � 177 reports Not RCTs 75 reports excluded: � 9 Factorial trials � 5 Non-inferiority trials � 10 Equivalence trials 102 reports � 20 Cluster Trials (107 RCT reports since 5 reports document two RCTs in one report. ) � 19 reports not RCTs � 1 Cross-over trial � 11 Too difficult to extract or not enough information provided PSI Conference, London, May 2017 9

General Results Volume Year Total Number of Reports Included RCTs 20 2016 95 20 19 2015 102 19 18 2014 71 12 17 2013 61 11 16 2012 49 8 15 2011 45 6 14 2010 60 8 13 2009 62 10 12 2008 36 2 11 2007 53 3 10 2006 50 8 684 107 Total PSI Conference, London, May 2017 10

Elicitation Categories • 7 categories based on DELTA publication (Cook et al. 2014) [1] – Anchor method – Distribution method – Health Economic method – Opinion-seeking method – Pilot-study method – Review of evidence base method – Standardised effect size method PSI Conference, London, May 2017 11

Study Characteristics Most common • Clinical Area – Mental Health (16. 7%) • Setting – Hospital (51%) • Elicitation Methods – Review of Evidence Base (46%) PSI Conference, London, May 2017 12

General Results PSI Conference, London, May 2017 13

General Results PSI Conference, London, May 2017 14

General Results PSI Conference, London, May 2017 15

General Results Shows that most common method across all the clinical areas is using previous research PSI Conference, London, May 2017 16

General Results Using DELTA categories, most common method is consistently Review of Evidence PSI Conference, London, May 2017 17

Estimated/Target Standardised Effect Size Smallest Target Effect sizes Largest Target Effect sizes PSI Conference, London, May 2017 18

Estimated/Target Standardised Effect Size Smallest Target Effect sizes Seems very large Largest Target Effect sizes PSI Conference, London, May 2017 19

Estimated/Target Standardised Effect Size Smallest Target Effect sizes Large variation Largest Target Effect sizes PSI Conference, London, May 2017 20

Estimated vs Observed Standardised Effect Size General relationship Estimated SES are higher than Observed PSI Conference, London, May 2017 21

Estimated vs Observed Standardised Effect Size General relationship Estimated SES are higher than Observed. As predicted PSI Conference, London, May 2017 22

Estimated vs Observed Standardised Effect Size Very high observed effect size Renal/Urology study. p<0. 001 PSI Conference, London, May 2017 23

Limitations • Only one reviewer, 177 reports to read through and 102 to extract information from. • No QA available • Some reports difficult to extract from due to unclear sample size justifications • In order to compare effect sizes for clinical categories, categories may need to be combined. This needs consultation with a clinical expert. PSI Conference, London, May 2017 24

Summary • Mental Health is most commonly published clinical area for RCTs. • Most common method of elicitation is using previous research or a combination of various methods including a review of the evidence. (49%) • Target effect sizes are typically larger than observed effect sizes (this is as expected) PSI Conference, London, May 2017 25

References 1. JA Cook, J Hislop, TE Adewuyi, K Harrild, DG Altman, CR Ramsay, C Fraser, B Buckley, P Fayers, I Harvey, AH Briggs, JDNorrie, D Fergusson, I Ford, LD Vale. Assessing methods to specify the target difference for a randomised controlled trial: DELTA (Difference ELicitation in Tri. Als). Health Technology Assessment 18(28), 2014 PSI Conference, London, May 2017 26

? ? ? Thank you for your attention. Any Questions? ? PSI Conference, London, May 2017 27