Getting to grips with evidence that matters NIHR

Getting to grips with evidence that matters! NIHR Nottingham Hearing Biomedical Research Unit, UK Melanie Ferguson Helen Henshaw British Society of Hearing Aid Audiologists 13. 9. 14

Learning outcomes • To explain the hierarchy of evidence • To describe how to assess quality of research articles Wong and Hickson, 2012, Evidence-based practice in Audiology, Plural Publishing

Pie charts

Hierarchy of evidence Cox (2005) J Am Acad of Audiology, 16 (7) Str en gth of evi den ce AAA (2010) APD guidelines

Level of evidence – validity criteria Basis of systematic review on auditory training (Henshaw and Ferguson, 2013) Scientific study-specific Intervention-specific • Randomisation? • • Power calculation? – For sample size (n) Ecologically valid training environment (at home)? • Training performance feedback? • Follow-up assessment? • -----Compliance • • Blinding of participants and researchers? -----Outcome measure – selection – reporting 0 = flawed or no information form which to make a judgement 1 = weak information or lack of detail 2 = appropriate use and reporting

Table 1: Study quality scores. Scientific study validity criteria Article Paper #1 Randomisation? Power calculation to determine sample size? Blinding of participants and researchers? 0 1 2 Intervention-specific study validity criteria Follow-up to Ecologically Training examine valid (atperformance retention of home) training feedback training environment? provided? effects? 0 1 2 Study quality score Level of evidence* 2 very-low Comments Paper #2 0 1 2 0 1 2 10 high Scoring: please circle 0 for flawed or no information from which to make a judgement, 1 = weak information or lack of detail, 2 = appropriate use and reporting, for scientific and intervention specific validity criteria. *Level of evidence (PTO): Study quality score of 0 -3 = very low, 4 -6 = low, 7 -9 = moderate, 10 -12 = high (adapted from GRADE Working Group, 2004).

Interactive session • Two papers – abstract, introduction, methods - read • Take table headings (e. g. randomisation) – Search paper – Make your judgement of quality (0, 1, 2) 0 = flawed or no information form which to make a judgement 1 = weak information or lack of detail 2 = appropriate use and reporting • Do same with second paper • Discussion at the end Rubbish Somewhere in between Good

Example - randomisdation • Paper #1 • Pg 920 – “All the participants were submitted to the evaluation only after they were assigned to the Experimental group and Control group, and the individuals themselves pick a number to be randomized to which group they would be sent to”.

Level of evidence Study quality score Level of evidence Confidence in estimation of effect 0 -3 Very low The estimation of effect is uncertain Low Further evidence is very likely to impact on our confidence in the estimation of effect and are likely to change the estimate 7 -9 Moderate Further evidence is likely to impact on our confidence in the estimation of effect and may change the estimate 10 -12 High Further evidence is very unlikely to change our confidence in the estimation of effect 4 -6 From GRADE working group 2004 End of session – what do you think the quality of the papers is

Purpose of the exercise • Brief introduction to appraisal of the literature • To highlight the factors that underlie “quality” of a paper to gain a broad understanding of what quality means Stop talking Get stuck in

Table 1: Study quality scores. Scientific study validity criteria Article Randomisation? Power calculation to determine sample size? Blinding of participants and researchers? Intervention-specific study validity criteria Follow-up to Ecologically Training examine valid (atperformance retention of home) training feedback training environment? provided? effects? Study quality score Level of evidence* Comments Randomisation not clear No power calculation Paper #1 Double-blinding? Participants are not able to be blinded (intervention vs. no intervention) 0 1 2 0 1 2 2 very-low Training completed in lab No feedback mentioned No follow-up assessmentioned General point: not repeatable because outcome measures are not clear/referenced. Minimisation – best form of randomisation – adaptive stratified sampling that is used in clinical trials - aims to minimise the imbalance between participants in two groups based on pre-specified factors. (Pocock & Simon, 1975) Paper #2 0 1 2 0 1 2 10 high Power calculation = 20 individuals per group to detect 2. 5 d. B SNR difference in digit triplets between the groups, Cohen’s d =. 89 No blinding mentioned Training took place @ home Feedback (correct/incorrect response) was provided during training and at the end of each session Follow-up 4 weeks post-training Scoring: please circle 0 for flawed or no information from which to make a judgement, 1 = weak information or lack of detail, 2 = appropriate use and reporting, for scientific and intervention specific validity criteria. *Level of evidence (PTO): Study quality score of 0 -3 = very low, 4 -6 = low, 7 -9 = moderate, 10 -12 = high (adapted from GRADE Working Group, 2004).