CASP Critical Appraisal Skills Programme making sense of

CASP Critical Appraisal Skills Programme making sense of evidence

CASP Competing factors and pressures Expectations Evidence Opinions Financial pressures Legal pressures Experience Political pressures Time pressures

CASP Types of evidence • your own experience • colleagues’ or experts’ opinions • published evidence (in books, journals, reports, on the internet etc. . . ) • unpublished evidence (local studies, studies that are never submitted/accepted for publication)

CASP What is evidence-based practice? “when we intervene in the lives of others we should do so on the basis of the best evidence available regarding the likely consequences of that intervention” G Macdonald, 1998

CASP Publication bias Papers with "interesting" results are (or may be) more likely to be: • submitted for publication • accepted for publication • published in a major journal and in English Language • quoted by authors • quoted in newspapers

CASP Pink Paint Programme

CASP Robust randomisation procedures - to ensure that the variables are equal in both groups - to minimise bias - to ensure that the results are generalisable

CASP Randomised controlled trial intervention group 1 Outcome group 2 Outcome population control

CASP Trial Quality - Does it matter? I. Randomisation

CASP Blinding = participants don’t know what intervention they are getting Double blinding = those giving the intervention don’t know what the participant is receiving

CASP Trial Quality - Does it matter? Blinding

CASP Loss to follow-up It is important to ensure that all those that are randomised into the trial are followed up to the trial’s conclusion

CASP Loss to follow-up Total participants = 283 Completed > 80% = 153 Found at follow-up = 95 Sufficient data to analyse = 92 32% of original group Completed < 80% = 130 Lost to follow-up = 58 Insufficient data to analyse = 3 Edleson and Syers, 1990

CASP Intention to treat analysis

CASP Apparently low risk patients eligible for trial Intensive care Key: Low Medium High Usual care

CASP Patients do not get treatment they are allocated. Intensive care Usual care

CASP Strategies to deal with this 1. Reject trial as spoilt 2. Exclude patients who did not get right treatment (comparing the outcomes only for those people who got the treatment they were supposed to) 3. Analyse according to the treatment people actually got 4. Pretend people got the treatment they were supposed to and analyse results comparing randomised groups

CASP Strategy 2 - exclusion Intensive care Usual care

CASP Strategy 3 - care received Intensive care Usual care

CASP Strategy 4 - Intention-to-treat Intensive care Usual care

CASP Intention to treat analysis Analysing people, at the end of the trial, in the groups to which they were randomised, even if they did not receive the intended intervention.

CASP Trial Quality - Does it matter? Intention-to-Treat Analysis

CASP Kinds of study design SINGLE STUDIES Experimentald trials controlle d e is m o d n a r • Quasi- Qualita ti exp • quasi -random erimental ised co ntrolled trials REVIEWS OF STUDIES Systematic reviews Meta-analysis ve l Pre-e ta n e m i r xpe t tests s o p d n • pre a rt studies • coho rol studies ont c e s a • c

CASP The best study for our questions

CASP Systematic Review A review of all the literature on a particular topic, which has been systematically identified, appraised and summarised giving a summary answer.

CASP Are the Results Important? • Outcome measures – what is their clinical significance? – Discrete vs continuous measures • Statistical significance – could these results have been obtained if there was really no difference between the groups?

CASP Framing effects - 1 Mammography A. Relative risk reduction 34% B. Absolute risk reduction 0. 06% C. Number needed to treat (NNT) 1592 D. Percentage of event-free patients 99. 82% vs 99. 8% Fahey et al BMJ 1995 1

CASP Framing effects - Cardiac Rehabilitation 2 A. Relative risk reduction 20% B. Absolute risk reduction 3% C. Number needed to treat (NNT) 31 D. Percentage of event-free patients 84% vs 87% Fahey et al BMJ 1995 1

CASP Framing effects • Relative risk reduction Cardiac rehab. 20% Mammography 34% • Absolute risk reduction 3% 0. 06% • Percentage of event-free patients 84% vs 87% 99. 82% vs 99. 8% • Number needed to treat (NNT) 31 1592

CASP The Number Needed to Treat • The number of patients who need to be treated to prevent one bad outcome – Requires discrete outcomes (event rates) – Relates directly to patients, unlike relative measures • NNT = 1 / (absolute risk difference)

CASP Results Mean = the “average” of all individual results Median = the mid point; the middle result when all data values are put in sequential order Mode = the most frequently occurring value or category

CASP Results 116 120 122 127 130 138 140 142 145 146 150 155 158 Mean = 2055 = 137 15 Median = 139 Mode = 127

CASP Results

CASP Results Mean = 137

CASP Results 100 110 120 130 - 1 s. d. - 2 s. d. 140 150 + 1 s. d. + 2 s. d. Mean = 137 160 170

CASP The p-value in a nutshell Could the same result have occurred by chance if the intervention actually had no effect? The result is likely to be due to chance The result is unlikely to be due to chance 0 1 p < 0. 05 p > 0. 05 a statistically significant result not a statistically significant result p = 0. 05 1 20 or 1 in 20 result fairly unlikely to be due to chance p = 0. 5 1 2 or 1 in 2 result quite likely to be due to chance

CASP Generalisability • Despite the size and duration of this trial, the populations of patients with OA and RA are much larger and therapy continues for substantially longer than 6 months. Moreover, many patients with OA and RA have comorbid illnesses…that would have excluded them from the current study, Consequently, the results of this study do not address the occurrence of rare events, nor can they be extrapolated to all patients seen in general clinical practice. Silverstein et al 2000

CASP The challenge • do people want to use evidence of effectiveness and can this fit into the culture of social services? • are people ready to be challenged? - evidence may say that favourite or common practices are not effective • do people have access to evidence of effectiveness? • do people have time to search for and read research papers? • can people understand research papers that they read? • can people make use of the evidence they get from research papers and act on it?

CASP Is the structure appropriate and will it stand up to being challenged?

CASP Critical appraisal: questions to apply to trials Is the trial trustworthy? validity What does the trial tell you? results Will it help? relevance

CASP Confidence Interval Is the range within which the true size of effect (never exactly known) lies, with a given degree of assurance (95% or 99%).

CASP Confidence Intervals (Wobble factor)

CASP Confidence Interval (CI) = the wobble factor, how sure are we about the results? - the shorter the CI the more certain we are about the results - if it crosses the line of 1 (no treatment effect) the intervention might not be doing any good and could be doing harm

CASP Reduction in mortality post MI 0. 5 1960 1 RCT Odds Ratio 1 2 23 Patients Treatment Control Antman JAMA 92

CASP Reduction in mortality post MI Cumulative Year RCTs 1960 1 1965 2 3 4 7 Pts 0. 5 23 65 149 316 1793 Odds Ratio 1 2 Treatment Control Antman JAMA 92

CASP Reduction in mortality post MI Cumulative Year RCTs 1960 1 1965 1970 1975 1980 1985 1990 2 3 4 7 10 11 15 17 22 23 27 33 65 70 Pts 0. 5 23 65 149 316 1793 2544 2651 3311 3929 5452 5767 6125 6571 47185 48154 Odds Ratio 1 Textbook 2 Recommendations Rout Specif Exp NOT Treatment 1 1 2 p < 0. 01 p < 0. 00001 Control 5 15 6 1 1 1 2 8 1 8 7 2 21 5 10 2 8 8 12 4 3 1 1 Antman JAMA 92