Quantitative Methods Reliability Validity and Use of SEM

Quantitative Methods: Reliability, Validity, and Use of SEM to Assess Psychometric Equivalence Ron D. Hays, Ph. D. UCLA Department of Medicine Los Angeles, CA November 18, 2005 1

Measurement Range for Health Outcome Measures Nominal Ordinal Interval Ratio 2

Indicators of Acceptability u u u Response rate Administration time Missing data (item, scale) 3

Variability u u All scale levels are represented Distribution approximates bell-shaped “normal” 4

Measurement Error Observed = true + systematic + random score error (bias) 5

Flavors of Reliability u u u Test-retest (administrators) Intra-rater (raters) Internal consistency (items) 6

Intraclass Correlation and Reliability Model Reliability Intraclass Correlation One-Way Two-Way Fixed Two-Way Random 7

Cronbach’s Alpha Source df SS MS Respondents (BMS) 4 11. 6 2. 9 Items (JMS) 1 0. 1 Resp. x Items (EMS) 4 4. 4 1. 1 Total 9 16. 1 Alpha = 2. 9 - 1. 1 = 1. 8 = 0. 62 2. 9 8

Reliability Minimum Standards u 0. 70 or above (for group comparisons) 0. 90 or higher (for individual assessment) u » SEM = SD (1 - reliability)1/2 9

Construct Validity u Does measure relate to other measures in ways consistent with hypothesis? u Responsiveness to change 10

Responsiveness to Change and Minimally Important Difference u HRQOL measures should be responsive to interventions that changes HRQOL Evaluating responsiveness requires assessment of HRQOL u » pre-post intervention of known efficacy » at two times in tandem with anchor » HRQOL change among people who changed on anchor 11

Self-Report Anchor u. Overall has there been any change in your asthma since the beginning of the study? Much improved; Moderately improved; Minimally improved No change Much worse; Moderately worse; Minimally worse 12

Clinical Anchor u“changed” group = seizure free (100% reduction in seizure frequency) u“unchanged” group = < 50% change in seizure frequency 13

Responsiveness Indices (1) Effect size (ES) = D/SD (2) Standardized Response Mean (SRM) = D/SD† (3) Guyatt responsiveness statistic (RS) = D/SD‡ D = raw score change in “changed” group; SD = baseline SD; †SD = SD of D; ‡ SD = SD of D among “unchanged” 14

Effect Size Benchmarks u Small: 0. 20 ->0. 49 u Moderate: 0. 50 ->0. 79 u Large: 0. 80 or above 15

Hypothetical Multitrait/Multi-Item Correlation Matrix 16

Confirmatory Factor Analysis u Compares observed covariances with covariances generated by hypothesized model u Statistical and practical tests of fit u Factor loadings u Correlations between factors u Regression coefficients 17

Fit Indices • Normed fit index: 2 null - 2 model 2 null 2 2 null df null • Non-normed fit index: - model df model null 2 df null • Comparative fit index: 1 - 2 model - df - 1 model 2 null - dfnull 18

19

20

Acknowledgments u. Supported in whole by UCLA Center for Health Improvement in Minority Elders/Resource Centers for Minority Aging Research, National Institute on Aging (AG-02 -004) 21