Lecture 6 Reliability and validity of scales cont

Lecture 6: Reliability and validity of scales (cont) 1. In relation to scales, define the following terms: - Content validity - Criterion validity (concurrent and predictive) - Construct validity (discriminant and convergent) - Responsiveness 2. Use of proxy respondents

Validity • Depends on purpose: – screening: discrimination – outcome of treatment: responsiveness, sensitivity to change – prognosis: predictive validity

Content and face validity • Judgment of “experts” and/or members of target population • Does measure adequately sample domain being measured? • Does it appear to measure what it is intended to measure? (eyeball test)

Criterion validity • Criterion (“gold” standard) • Concurrent criterion validity – e. g. , screening test vs diagnostic test • Predictive criterion validity – e. g. , cancer staging test vs 5 -year survival

Construct validity • Is theoretical construct underlying the measure valid? • Development and testing of hypotheses • Requires multiple data sources and investigations: – Convergent validity: measure is correlated with other measures of similar constructs – discriminant validity: measure is not correlated with measures of different constructs

Responsiveness of measures • Ability to detect clinically important change over time or differences between treatments • Requirement of evaluative measures • Two approaches: – external responsiveness (validation against change in external criterion) – internal responsiveness: compute effect size of ratio of change score to measure of variability (different formulae)

Validity of Delirium Index (DI) • Convergent validity – DI will be correlated with measures of current health/function: • Current ADL disability (Barthel Index) • Current severity of illness (clinical judgment and physiological severity score from APACHE)

Validity of Delirium Index (DI) • Discriminant validity: – DI will be weakly correlated with measures of previous health/function: • premorbid level of ADL disability • severity of dementia • comorbidity (prior)

Validity of DI (cont) • DI score will predict probability of one-year survival • DI score will be sensitive to: – changes in medication exposures – changes in environmental factors

Spearman correlation coefficients between Delirium Index and 3 baseline measures of current status

Spearman correlation coefficients between Delirium Index and 3 baseline measures of prior status

Delirium severity and survival • Proportional hazards regression of delirium severity in delirium cohort • Mean of 1 st 2 DI scores • Results – significant interaction: DI predicted survival in patients with delirium alone, not in those with dementia

Effects of medications and environmental factors • Repeated in-hospital measures of DI, medications (medical record), and environmental factors (using checklist) • Repeated measurements analysis performed to investigate associations of changes in risk factors with change in DI score.

Environmental risk factors for delirium Source: Mc. Cusker et al, JAGS 2001, 49: 1327 -34

Medications and DI • Medication changes (increase in # medications, use of anticholinergic medications) were associated with increase in DI score.

Responsiveness of DI • Internal responsiveness: ability to capture real change over time – effect size (change in mean DI/SD at baseline) – standardized response mean (change in mean DI/SD of change) • External responsiveness: are changes in the DI correlated with other measures of change?

Internal responsiveness of the Delirium Index (DI) at 8 weeks

Validity of DI: Conclusions • In patients with delirium +/- dementia: – Correlated with measures of current function – Responsive to change over time – Responsive to changes in environmental factors and medications • In patients with delirium alone: – Predicts survival

Example: Inflammatory Bowel Disease Questionnaire (IBDQ) • Disease-specific HRQo. L measure • 30 items covering 4 domains – bowel symptoms – systemic symptoms – emotional function – social function • How to validate?

Validation of Inflammatory Bowel Disease Questionnaire • Questionnaires administered to 42 patients with inflammatory bowel disease on 2 occasions, 1 month apart: – IBDQ – Global ratings of function – Global ratings of change by physician and a relative – Disease Activity Index – Emotional function scale from generic HRQo. L instrument

Validation of IBDQ: predictions and results • Correlation of 0. 5 or more between patients’s global rating of change and IBDQ subscale on bowel symptoms (result: 0. 42) • Correlation of 0. 3 or more between change in Disease Activity Index and IBDQ subscale on bowel symptoms (result: 0. 33) • Correlation of 0. 3 or more between change in Disease Activity Index and IBDQ subscale on systemic symptoms (result: 0. 04) • Correlation of 0. 5 or more between change in generic measure of emotional function and IBDQ subscale on emotional function (result: 0. 76)