Simultaneous Comparisons of Accuracy Sensitivity and Specificity in

Simultaneous Comparisons of Accuracy, Sensitivity, and Specificity in Diagnostic Studies with Binary Data Jörg Kaufmann, Ph. D. , Schering AG, Berlin, Germany Suming Chang, Ph. D. , Berlex, New Jersey, USA

Issues which will be addressed l Choice of efficacy measures for the primary endpoint l Multiple evaluations on each patient for the primary efficacy analysis Page 2

Issues which will not be addressed l Imperfect gold standard l Establishing a gold standard l Study design – parallel group, cross-over l Most appropriate image set comparison: Page 3 Ø Post versus pre Ø Post + pre versus pre Ø More than 1 post sequence

More issues which will not be addressed l Approaches to handling uninterpretable or non -assessable images or regions of interest in images l Disease state not binary l Multiple blinder readers and clinical investigator evaluations l By the time the trial is done, technology will have changed Page 4

Data Structure in Diagnostic Imaging Studies Page 5

Two way design Strata Non diseased Diseased T 1 T 2 P 01 P 02 (n 01) (n 02) P 11 P 12 (n 11) (n 12) nsi number of patients/diagnostical units psi proportion of correct results n 0 i ≠ n 1 i in general Clustered data: each patient contributes diseased and non-diseased vessel segments (observational units, correlated) Paired design: each patient received both treatments, T 1 and T 2 correlated Page 6

Indices for Validity Specificity effect in non diseased patients Sensitivity effect in diseased patients T 1 T 2 P 01 P 02 P 11 P 12 T 2 – T 1 Accuracy (overall effect across strata) Psi proportion of correct results in stratum s on treatment i ws is a weight assigned to stratum s, ∑ ws =1 Page 7

Four Commonly Used Weighting Schemes Page 8

Comments on different weighting schemes Page 9 Pool • • CMH • Strata with lager number of patients/vessel segments get more weights • Weights generally yield an approximately unbiased estimate of d • Estimate has a potentially larger variance Prevalence Dependent Definition for accuracy in the regulatory guidelines Usually results in decreased power for inference Simpson’s paradox possible EQUAL • Corresponds to the accuracy index (area under the curve) in the ROC methodology for dichotomous data • Prevalence Independent INVAR • Usually yields an estimate of d with minimum variance. • Weights involve unknown nuisance parameters (estimated from the data, addition random variability)

Example: D. V. Mehrotra (6)pp 1349 Strata Page 10 T 1 T 2 Non diseased 0. 48 0. 530 0 48/100 53/100 Diseased 0. 8 0. 95 1 48/60 57/60 Weighting Scheme W 0 W 1 T 1 w T 2 w w P-value Pool 0. 625 0. 375 0. 688 0. 082 CMH 0. 625 0. 375 0. 688 0. 082 EQUAL 0. 50 0. 64 0. 74 0. 1 0. 031 INVAR 0. 41 0. 59 0. 669 0. 778 0. 109 0. 018

Overall Effect across Strata 1. 00 T 2 0. 90 P 12= 0. 95 T 1 0. 80 P 11= 0. 80 0. 70 0. 60 0. 50 T 1 T 2 P 02= 0. 53 P 01= 0. 48 0. 40 non diseased population CMH EQUAL INVAR total population Page 11 Diseased population

Closed testing with a-priori ordered hypotheses First step: compare the treatments in the total study population T 1 vs T 2 Accuracy across both strata (diseased and non-diseased) (CMH or EQUAL weights for each strata) Proceed to step 2 only if significant result obtained in step 1 Second step: compare the treatments in each sub-population T 1 vs T 2 Accuracy in diseased patient ~ Sensitivity Accuracy in non-diseased patient ~ Specificity Test all 3 hypotheses at the α = 0. 05 level. The closed testing procedures consider the multiplicity aspects. Page 12

Conclusions l Use statistical methods accounted for clustered data l Pre-specify the weighting scheme for the overall accuracy across strata l Pre-specify closed testing principle with a priori ordered hypotheses l Define all above in the study protocol prospectively l Recommend to communicate with the authorities in the early trial design phase Page 13

Thank you for your attention Accuracy in total study population n ty i i v i sit d Sen sease di ient pat in y t i c cifi ased e p S e dis n no atient p Experim Observa ental un tional u it? nit?

REFERENCES 1. Biesheuvel Eghart H. E. , Hothom L. A. : Protocol designed subgroup analyses in multiarmed clinical trials: multiplicity aspects. Journal of Biopharmaceutical Statistics 2003, 13. No 4, pp 663673 2. Kaufmann J. , Werner C. , Brunner E. : Nonparametric methods for analysing accuracy of diagnostic test with multiple readers. Statistical Methods in Medical Research 2005, 14: 129 -146 3. Kaufmann J. , Koch G. G. : Statistical considerations in the design of clinical trials, weighted means and analysis of Covariance. Proceedings of the Conference in Honor of Shayle Searle, Cornell University , Biometrics unit 1996, p 77 -92 4. Liu Aiyi, Schisterman E. F. , Mazumdar M. , Hu Jiong: Power and sample size calculation of comparative diagnostic accuracy studies with multiple correlated test results, Biometrical Journal 2005, 47, 2; 140 -150 5. Maurer W. , Hothom L. A. , Lehmacher W. : Multiple comparisons in drug clinical trials and preclinical assays: a priori ordered hypotheses; Biometrie in der chemisch-pharmazeutischen Industrie 1995, 6; 3 -18; Gustav Fischer Verlag 6. Mehrotra Devan : Stratification issues with binary endpoitns; Drug Information Journal 2001, 35; 1343 -1350 7. Mehrotra Devan , Railkar R. : Minimum risk weights for comparing treatments in stratified binomial trials, Statistics in Medicine 2000, 19; 811 -825 8. Scott P. E. and Campbell G. : Integration of subgroup analyses in medical device clinical trials; Drug Information Journal 1988, 213 -220 9. Shayle R. Searle: Linear models for unbalanced data, John Wiley & Sons, 1987 10. Swets John A. , Pickett Ronald M. : Evaluation of Diagnostic System-Methods from Signal Detection Theory; Academy Press 1982 Page 15