Do we need new statistical methodology Ian White

Do we need new statistical methodology? Ian White <ian. white@ucl. ac. uk> MRC Clinical Trials Unit at UCL Network for Applied Statisticians in Health (NASH) 21 st May 2018 MRC Clinical Trials Unit at UCL

What is NASH? • John Nash (English architect, 1752 – 1835) • Ogden Nash (American poet, 1902 – 1971) There was an old man of Calcutta, Who coated his tonsils with butter; Thus converting his snore Hypothesis generating From a thunderous roar needs randomised evaluation To a soft oleagenous mutter. 2 • Russian: Наш = “ours” MRC Clinical Trials Unit at UCL

Do we need new statistical methodology? • It depends 3 MRC Clinical Trials Unit at UCL

Statement of interest • I’m a methodologist − interested in trials, observational studies & metaanalysis − esp. missing data, treatment changes in trials, network meta-analysis • MRCCTU’s mission: − carrying out challenging and innovative studies − developing and implementing methodological advances in study design, conduct and analysis 4 MRC Clinical Trials Unit at UCL

What do we want? • Do our jobs more − effectively − efficiently − pleasantly − fulfillingly − profitably • Do we need new statistical methodology for this? − it depends on what role we are playing 5 MRC Clinical Trials Unit at UCL

Roles: the “can’t-do” statistician • You can’t interpret P>0. 05 as “no association”! • You can’t publish that association without describing all the other associations you also explored! (in an observational study) • You can’t replace non-complying participants with new ones! (in a trial) • You can’t say the treatment only works in a subgroup without doing an interaction test! (in a trial) OK from time to time, but not as a permanent mindset. What do we need for this role? • sound grasp of existing methodology • thick skin 6 MRC Clinical Trials Unit at UCL

Roles: the “can-do” statistician • How can I explore this aspect of my data? − how can I model children’s growth through puberty? − my data are missing not at random, what should I do? − some trial participants received rescue medication, what should I do? • How can I design my study? − should I run this trial as a factorial design? 7 What do we need for this role? • Often involves clever use of existing methodology • May need tweaking / extending existing methodology • May need really new methodology − usually not used in an applied paper MRC Clinical Trials Unit at UCL

Roles: defending our work • Against the “can’t-do” reviewer − “that analysis wasn’t pre-specified so you mustn’t report it” • Against the “can-do (and therefore should do)” reviewer − “you have missing data, you must use multiple imputation” What do we need for this role? • Good understanding of the methods • Good understanding of theory 8 MRC Clinical Trials Unit at UCL

Is some new methodology useful? • Certainly! • e. g. − intention-to-treat analysis − multiple imputation − network meta-analysis 9 MRC Clinical Trials Unit at UCL

Is all new methodology useful? • Certainly not! • Much new methodology has zero impact − why not? 10 MRC Clinical Trials Unit at UCL

What do applied statisticians need before we start using a new methodology? 11 • Valid theory? − less important than we might think (e. g. MICE) • Published use? − in more than one data set • Generality? − e. g. allows both categorical and continuous covariates; handles missing data • Simulations? − showing good performance − comparing new method with standard alternatives − but how do we know if they apply to our setting? • Useable software? − almost essential How do we decide? MRC Clinical Trials Unit at UCL

How do we judge when a new methodology is ready for our use? • Probably not very scientifically • It’s useful to understand how well developed a methodology is • I believe we can identify phases of methodological development − like phases of drug development − but not formalised 12 MRC Clinical Trials Unit at UCL

Phases of methodological development 1. Technique is used by methodological statistician with software purpose-built for the application − “First-in-data” − publication: e. g. Biostatistics 2. Technique is developed as a methodological project for use in new data sets − publication: e. g. Stat Med 3. Technique is used by applied statisticians in clinical publications − publication: e. g. AJE, BMJ − software available 4. Review articles promote use of technique; referees start to demand its use − publications in clinical or statistical journals 13 MRC Clinical Trials Unit at UCL

Phases of methodological development: multiple imputation Phase Publication 1: Technique is used by methodological statistician with software purpose-built for the application Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York: John Wiley and Sons; 1987. 2: Technique is developed as a methodological project for use in new data sets Rubin DB. Nested multiple imputation of NMES via partially incompatible MCMC. Stat Neerl. 2003; 57: 3– 18. 3: Technique is used by applied statisticians in clinical publications Horton NJ, Lipsitz SR. Multiple imputation in practice: Comparison of software packages for regression models with missing variables. Am Stat. 2001; 55: 244– 54. 4: Review articles promote use of technique; referees start to demand its use Sullivan TR, White IR, Salter AB, Ryan P, Lee KJ. Should multiple imputation be the method of choice for handling missing data in randomized trials? Stat Methods Med Res. 2016 MRC Clinical Trials Unit at UCL 14

Can new methodology be harmful? • Statistics is an unusual science in that we don’t just have to use a valid technique, we have to defend our choice − to avoid accusations of cherry-picking / datadredging • So new methodology can create confusion − especially if claims made for it are exaggerated 15 MRC Clinical Trials Unit at UCL

Arm-based network meta-analysis 16 • New proposal (~2014): change from modelling contrasts (asymmetrically) to modelling arms (symmetrically) − e. g. Lin L, Zhang J, Hodges JS, Chu H. Performing Arm-Based Network Meta-Analysis in R with the pcnetmeta Package. J Stat Softw. 2017; 80(5). − claimed to have better missing data properties & to lead to more appropriate summaries − can give quite different results from standard contrast-based methods − critics argue that it “breaks” randomisation • Consequences for applied network meta-analysis are unclear • Lots of debate in methodological fora − has debate been useful? MRC Clinical Trials Unit at UCL

Hierarchy for evaluating the effects of a diagnostic technique [if time] Here described for magnetic resonance imaging (MRI): • Technical performance: Does MRI result in good quality images which are anatomically representative? • Diagnostic performance: Do the images allow accurate diagnoses to be made? • Diagnostic impact: Does MRI change diagnostic confidence and displace other investigations? • Therapeutic impact: Do the results of MRI contribute to planning and delivery of treatment? • Impact on health: Does the use of MRI contribute to the improved health of the patient? 17 Mackenzie R, Dixon AK (1995). Measuring the effects of imaging: An evaluative framework. Clinical Radiology 50: 513 -518. MRC Clinical Trials Unit at UCL

Hierarchy for evaluating the effects of a diagnostic / statistical technique? Imaging technique Statistical technique Technical Good quality images performance Results are internally consistent Diagnostic Images allow accurate performance diagnoses Results are convincing to statisticians Diagnostic impact Images change diagnostic Results are convincing to confidence and displace clinicians other investigations Therapeutic impact Contributes to planning and delivery of treatment Results lead to change of treatment (compared to different analyses? ) Impact on health Contributes to the improved health of the patient Results lead to improved public health 18 MRC Clinical Trials Unit at UCL

Conclusions and questions • Sometimes we need new methodology • Sometimes new methodology is forced on us • We need new methodology to be carefully evaluated − need to be clear when methods are under development and when they are ready for practice • How can we judge when new methodology is ready for use? − should methodological journals rate their papers according to their readiness for practice? − do we need a methodological journal restricted to methods actually used in applied journals? − other suggestions? 19 MRC Clinical Trials Unit at UCL