CAT Critically Appraised Topic adapted from Sackett et

CAT (Critically Appraised Topic) (adapted from Sackett, et al. 2000) 1 -page summary of evidence resulting from critical appraisal of an article, test, etc. n Answers a specific foreground question n – “Compared to no treatment, does parent -administered treatment significantly improve the language skills of toddlers with language delay? ” C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 1

First part of CAT identical for tx and dx studies (see handout pp. 2 -3) Clinical bottom line: (appears 1 st but completed last) n Clinical question: n Search terms: n Appraised by whom, and date: n Synopsis of key (memorable) information, in a concise, maximally useful format (e. g. , types of subjects, procedures, measures, results, etc. ) C. Dollaghan, An Evidence-Based Approach to Clinical Practice n 5/28/04 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 2

CAT-egories (appraisal points) for a study of therapy (Sackett et al. , 2000) Prospective, controlled? n Random assignment? n Comparing > 2 conditions? n Recognizable subjects? n Evidence of pre-tx group similarity? n Blinding (insofar as possible) of evaluators, relevant others? n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 3

Appraisal points (cont. ) Control over nuisance variables? n Valid, reliable measures of tx effects? n Statistically significant difference (pvalue)? n Practically significant difference (d-value)? n Precision of treatment effects (narrow CI)? n Outcomes for all enrolled? C. Dollaghan, and An Evidence-Based Approach to Clinical Practice n 5/28/04 Cost-benefit feasibility analyses? 4 in Communication Disorders, Mc. Geachy Memorial Lecture, n University of Toronto

A sample treatment CAT n n n CAT: Language of delayed toddlers improves in response to parent-administered focused stimulation Clinical bottom line: Compared to an untreated control group, motivated mothers of low-vocabulary toddlers significantly decreased their speaking rate and language complexity and increased their vocabulary inputs in response to ~18 hr of instruction in focused stimulation techniques, and their children produced significantly more words and early grammatical forms. Clinical question: Compared to no treatment, does parent-administered treatment significantly improve the language skills of toddlers with language delay? Search terms: word learning AND toddlers, Pub. Med clinical query C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 5 Appraised by: Dollaghan in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Key appraisal points Prospective, controlled Yes n Randomized Yes n Comparing > 2 conditions Yes n Recognizable Ss Yes n Pre-tx similarity Yes n Blinding Yes Cn; no parent n Control over nuisance variables Yes n Valid, reliable measures Yes n Statistically significant differences Yes n Practically significant differences Yes n Precision of treatment effects No n Outcomes for all enrolled Yes n 5/28/04 Cost-benefit, analyses Yes 6 C. Dollaghan, feasibility An Evidence-Based Approach to Clinical Practice n in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Critical appraisal of evidence on diagnostic indicators The key variables by which individuals are identified as members of a class, ostensibly to improve prediction and outcome for them n Myriad diagnostic indicators have been proposed in communication sciences and disorders n Diagnostic indicators in your area of interest? C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 7 n in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Most diagnostic indicators in CSD are based on “Phase I” studies n Group mean comparison studies – People with, and people without, the condition of interest are compared with respect to a proposed indicator n Correlational studies – Association between proposed indicator and accepted indicators Such studies can’t address the two most crucial features of a diagnostic indicator: C. Dollaghan, An Evidence-Based Approach to Clinical Practice accuracy and precision 5/28/04 8 in Communication Disorders, Mc. Geachy Memorial Lecture, n University of Toronto

Accuracy and precision n Accuracy – The ability of an indicator to identify a condition of interest, i. e. , the amount of agreement between the proposed indicator and a reference standard n Precision – Width of confidence intervals (CI) for estimates of accuracy C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 9

Accuracy of a diagnostic indicator The ability of an indicator to identify a condition of interest, i. e. , the amount of agreement between the proposed indicator and a reference standard n Preferred measures of diagnostic accuracy: positive and negative likelihood ratios (Battaglia et al. , 2002) n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 10

Positive Likelihood Ratio (LR+) Reflects the degree of confidence that a person who scores in the positive (affected or disordered) range on a dx indicator does have the disorder n Formula: sensitivity/1 -specificity n The higher the LR+, the more informative the indicator for identifying people who have the disorder n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 11

Interpreting LR+ values (Sackett et al. , 1991) LR+ > 20 Very high; virtually certain that a person with this score has the disorder LR+ = 10 High; disorder very likely in a person with this score LR+ = 4 Intermediate; the indicator is suggestive of disorder but insufficient to diagnose LR+ = 1 Equivocal Equivoca ; a person who scores in the disordered range on the measure may or may not have the disorder; the measure provides no new information C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 12 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Negative Likelihood Ratio (LR-) Reflects the degree of confidence that a person scoring in the negative (normal) range on the diagnostic indicator truly does not have the disorder n Formula: 1 -sensitivity/specificity n The lower the LR-, the more informative the indicator for ruling out the presence of disorder n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 13

Interpreting LR- values (Sackett et al. , 1991) LR- < 0. 10 Very low ; virtually certain that a person scoring in this range does not have the disorder LR- = 0. 20 Low; disorder very unlikely LR- = 0. 40 Intermediate; the indicator is suggestive but insufficient to rule out the disorder LR- = 1. 0 Equivocal; a person scoring in the normal range on this measure may or may not be normal 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 14

Calculating sensitivity and specificity (nothing more than LR precursors) Sensitivity: the percentage of people with the disorder that the new indicator correctly classifies as disordered n Specificity: the percentage of people who don’t have the disorder that the new indicator correctly classifies as not disordered n The “true” status of every individual with regard to the disorder is established according to a gold (or reference) standard C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 15 n in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Disorder Status (re: Gold Standard) + Disorder (LI) New Test Result + Disorder (LI) a b c d -Disorder (LN) # with disorder 5/28/04 - Disorder (LN) # without disorder C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 16

Disorder Status (re: Gold Standard) + Disorder (LI) - Disorder (LN) + Disorder (LI) True positive New Test Result -Disorder (LN) a c False positive b d False negative True negative Sensitivity=a/a+c 5/28/04 (the proportion of people with the disorder that the new test identifies C. Dollaghan, An Evidence-Based Approach to Clinical Practice 17 in Communication Disorders, Mc. Geachy Memorial Lecture, as having the disorder) University of Toronto

Disorder Status (re: Gold Standard) + Disorder New Test Result -Disorder True positive - Disorder False positive a b c d False True Specificity = d/b+d negative (the proportion of people 5/28/04 without the disorder that the new test identifies as C. Dollaghan, An Evidence-Based Approach to Clinical Practice having the disorder) 18 in Communication Disorders, not Mc. Geachy Memorial Lecture, University of Toronto

Example 100 children diagnosed with language impairments (LI) and enrolled in language intervention, and 100 same-age children with no history of language impairment (LN), were administered a new test of grammatical morphology. n 80 of the children with LI, and 30 of the children with LN, scored in the disordered range on the new measure. n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 19

Disorder Status (re: Gold Standard) + Disorder (LI) - Disorder (LN) + Disorder (LI) 80 30 a b New Test Result -Disorder (LN) c d (20) 100 with 5/28/04 (70) 100 without disorder Sens= a/a+c= Spec = d/b+d = C. Dollaghan, An Evidence-Based Approach to Clinical Practice 20 80/100. 80 Memorial 70/100 in Communication Disorders, = Mc. Geachy Lecture, =. 70 University of Toronto

Why not just use sensitivity and specificity as measures of accuracy? It’s their interrelationship that is most important overall n Sensitivity and specificity vary substantially according sample characteristics, including N, base rate (prevalence), severity, confusability n Likelihood Ratios are not impervious to sample characteristics, but are much less affected than are sensitivity and specificity n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 21

Calculating Likelihood Ratios Sens =. 80 n Spec =. 70 n LR+ = sens/1 -spec =. 80/. 30 = 2. 67 n LR- = 1 -sens/spec =. 20/. 770 = 0. 29 n Several programs, some free on web, are set up to allow entry in 2 x 2 table format n In addition to accuracy measures, they also provide information on precision n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 22

Precision of a diagnostic indicator Width of confidence intervals (CI) for sensitivity, specificity, and likelihood ratios, calculated by adding and subtracting a multiple of standard error (e. g. , 1. 96 SE for a 95% CI) n Standard error depends on sample size and reliability; larger samples and higher reliability will result in narrower CIs, all else being equal n Sackett et al. (2000) appendix shows how to calculate CIs by hand, and programs (some free) provide CIs given raw numbers in a An Evidence-Based Approach to Clinical Practice 5/28/04 23 2 x 2 table. C. in. Dollaghan, Communication Disorders, Mc. Geachy Memorial Lecture, n University of Toronto

Sample size and precision: 95% CIs for studies with same LRs but different Ns Value Sens =. 80 Spec =. 70 LR+ = 2. 67 LR- = 0. 29 5/28/04 N = 200 (95% CI) (0. 71 -0. 87) (0. 60 -0. 79) (1. 98 -3. 70) (0. 19 -0. 42) N = 20 (95% CI) (0. 44 -0. 98) (0. 35 -0. 93) (1. 12 -7. 66) (0. 08 -0. 87) C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 24

CAT-ing evidence on a diagnostic indicator (Sackett et al. , 2000; Battaglia et al. , 2002) n Does the study report a comparison between measures, or measure and gold standard? – sine qua non for evidence of diagnostic accuracy n Was the gold (or reference) standard valid, reliable, and/or reasonable? – Gold standard and new indicator also must be independent to avoid incorporation bias that can inflate accuracy C. Dollaghan, An Evidence-Based Approach measures to Clinical Practice 5/28/04 25 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Criteria for diagnostic indicators (cont. ) Were patients enrolled prospectively and consecutively (or by random assignment), and n Did the sample include a spectrum of patient types and severities? n – These two criteria are important in avoiding spectrum bias, in which the sample includes only clear-cut or handpicked cases and thus does not represent diagnostic task. Practice C. Dollaghan, the An Evidence-Based Approach to Clinical 5/28/04 26 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Criteria for diagnostic indicators (cont. ) n n n Were the new measure and the reference standard administered independently, by different examiners, and Were the examiners blinded to the subject’s performance on the other test and to other relevant subject information? Were the new measure and the reference standard both administered to all subjects and controls? – Important to avoid differential verification bias, when controls are assumed to be normal without testing on gold standard C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 27

Criteria for diagnostic indicators (cont. ) n Do likelihood ratios suggest adequate diagnostic accuracy? – LR+ > 4. 0 (> 10 cf. Bayes Library, 2002) – LR- < 0. 40 (< 0. 20, cf Bayes Library, 2002) Precision (narrow confidence intervals)? n Feasibility for usual clinical practice? n Value (i. e. , better than current measure)? n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 28

Evidence on norm-referenced tests as diagnostic indicators for early LI Many norm-referenced tests have diagnosis of LI as their explicit purpose n A growing number of tests meet typical psychometric criteria, e. g. N = 100 subjects per age level; reliability >. 90; means, standard deviations, and standard errors of measurement n But very few provide evidence of diagnostic accuracy or precision, and none meet the C. Dollaghan, An Evidence-Based appraisal Approach to Clinical Practice recommended critical criteria 29 5/28/04 n in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Norm-referenced tests not providing information on accuracy or precision Test of Language Development (TOLD) n Sequenced Inventory of Language Development (SICD) n Test of Early Language Development (TELD) n Reynell Scales n Mac. Arthur Communicative Development Inventories (CDI) C. Dollaghan, An Evidence-Based Approach to Clinical Practice n 5/28/04 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 30

A few tests provide information allowing accuracy and precision to be calculated Age LI LN LR+ (95% CI) LR- (95% CI) PLS-4 Total language score < 85 3 24 24 6. 7 (2. 6 -19. 4) 0. 19 (. 08 -. 42) 4 23 23 18 (3. 6 -102) 0. 23 (. 10 -. 44) 5 28 28 4. 4 (2. 1 -10. 2) 0. 26 (. 12 -. 50) 3 -5 75 75 6. 7 (3. 7 -12. 5) 0. 23 (. 14 -. 35) CELF-P Total language score < 85 3 -5 80 80 5. 3 (2. 9 -10. 2) 0. 45 (. 34 -. 58) CELF-P Total language score < 77 3 -5 80 80 12. 7 (4. 4 -37. 8) 0. 54 (. 43 -. 66) But note that these studies would fail many of the other C. Dollaghan, criteria, An Evidence-Based Approach to Clinical Practice critical appraisal their accuracy notwithstanding. 5/28/04 31 in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

The situation is no better for other proposed diagnostic indicators Few compare indicator to a gold standard, so accuracy can’t be determined n Few used blinded examiners, so a high potential for context and other biases n Small samples, wide CIs (rarely provided) n When sensitivity and specificity have been reported, they have sometimes been calculated incorrectly and/or misinterpreted C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 32 n in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

I choose not to despair Knowing the limitations of our diagnostic tools is an important prerequisite to designing better diagnostic tools n Several possible ways forward, most involving clinician-researcher partnerships n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 33

A way forward to EBP in Speechlanguage pathology and Audiology n Designing studies to meet the criteria for strong evidence – e. g. , STARD (Bossuyt et al. , 2003) statement n Large-scale, cooperative studies of diagnostic indicators – CARE-COAD model (Straus et al. 2002) n Dealing with the absence of a gold standard – e. g. , Demissie et al. , 1998; Dunson, 2001; reliability and outcome studies n Diagnostic studies as multivariable, prediction research & Practice Grobbee, C. Dollaghan, An Evidence-Based(Moons Approach to Clinical 5/28/04 2002) in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 34

Test yourself Critical appraisal of diagnostic test (handout p. 5) n Critical appraisal of treatment study (handout p. 4) n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 35

Critical appraisal and CAT enable the remaining steps to EBP 5. Decide whether the evidence is strong enough to influence your clinical practice 6. Integrate the evidence with the “intangibles” 7. Update! 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 36

EBP is itself a set of assumptions, not a cult Ultimately, strong evidence will be needed to determine whether EBP results in improved clinical service. n And EBP can’t be applied blindly, to all kinds of problems. . . n 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 37

As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of C. Dollaghan, An Evidence-Based Approach to Clinical Practice 5/28/04 38 the parachute. in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto

Thanks! References 5/28/04 C. Dollaghan, An Evidence-Based Approach to Clinical Practice in Communication Disorders, Mc. Geachy Memorial Lecture, University of Toronto 39