Chapter 8 Reliability TestRetest Parallel Test Interrater and

Basics of Reliability and Related Definitions • Reliability: the extent to which a measurement is free from measurement error. • True score: the mean of an infinite number of measurements of a single person taken under identical circumstances. • The lower the measurement error, the better the instrument estimates the true score. • The larger the sample, the more errors in measurement tend to “cancel out. ” Copyright © 2016 Wolters Kluwer • All Rights Reserved 2

Test–Retest Reliability (Stability, Reproducibility) • Demonstrated by administering a measure to the same person on at least two occasions. • Maintaining anonymity can be an issue. • Differences between testing results may be caused by: • Subject attrition between testings. • Traits change over time; generally the longer the time gap, the lower the test-retest reliability. • Answers to PROs may be remembered and replicated if time gap is too short. • Boredom or annoyance when measured a second time may cause careless responses. • Rehearsal/learning effect (especially with tests of performance). • Regression to the mean with extreme scores. Copyright © 2016 Wolters Kluwer • All Rights Reserved 3

Parallel Test Reliability • Used when development of multi-item parallel tests (alternative-form tests) is desirable. • Parallel tests can be created by randomly selecting two sets of items from a tested item pool. • Useful when multiple measures are taken in a period of time and carryover effects need to be avoided. • A major source of measurement error in parallel test reliability is the sampling of items used on the alternate form. • Appropriate only for reflective multi-item scales. Copyright © 2016 Wolters Kluwer • All Rights Reserved 4

Interrater and Intrarater Reliability • A key source of measurement error can result from the person making observations or recording the measurements. • Interrater (or interobserver) reliability assessment involves having two or more observers independently applying the instrument with the same people and comparing scores for consistency. • Intrarater reliability assesses the consistency of the same rater measuring on two or more occasions, blinded to the scores he or she assigned on any previous measurements. • Actions to increase this type of reliability: • Developing scoring systems needing little inference. • Meticulous instructions with precise scoring guidelines and clear examples. • Training of scorers. Copyright © 2016 Wolters Kluwer • All Rights Reserved 5

Choosing a Type of Reliability to Estimate • Consider all possible sources of measurement error and then assess as many types of reliability as are meaningful. • Type of instrument will dictate the type of reliability to estimate; for example: • Reliability of a formative verbal report index should only be assessed through test–retest. • Interrater or intrarater reliability is essential for observational measures; other forms are situational. • Table 8. 2 in the text summarizes key features of different types of reliability that need to be demonstrated according to measurement type and major source of error. Copyright © 2016 Wolters Kluwer • All Rights Reserved 6

QUESTION Place the letter of the type of reliability listed in the left-hand column next to the term that best matches it in the right-hand column: Types of Reliability Related Terms A. Test–Retest ___ Used when multi-item tests are needed that measure same the construct. B. Parallel Test ___ Assesses responses from the same scorer at different times. C. Interrater ___ Stability, Reproducibility. D. Intrarater ___ Assesses responses from different scorers. Copyright © 2016 Wolters Kluwer • All Rights Reserved 7

ANSWER Place the letter of the type of reliability listed in the left-hand column next to the term that best matches it in the right-hand column: Types of Reliability Related Terms A. Test–Retest _B_ Used when multi-item tests are needed that measure same the construct. B. Parallel Test _D_ Assesses responses from the same scorer at different times. C. Interrater _A_ Stability, Reproducibility. D. Intrarater _C_ Assesses responses from different scorers. Copyright © 2016 Wolters Kluwer • All Rights Reserved 8

Intraclass Correlation Coefficient as a Reliability Parameter • A reliability coefficient indicates how well people can be differentiated from one another on the target construct despite measurement error. • True score variance is never known but can be estimated based on variability between people. • The reliability coefficient is calculated by means of the intraclass correlation coefficient, which can be used when a measure yields continuous scores. • The basic assumptions are that the scores for different people being measured: • Are independent and normally distributed. • Are randomly selected. • That the residual variation (error) is random and independent, with a mean of zero. Copyright © 2016 Wolters Kluwer • All Rights Reserved 9

ICC Models for Fully Crossed Designs • Score variability can be conceptualized as being of two types: • Variation from person to person being measured. • Variation from measurement to measurement of each person for k measurements. • The term fully crossed means that each person is rated by k raters (or completes a measure k times), and each rater rates everyone. • The value for k is often 2: a test and then a retest, or two observers’ ratings. • The two-way ANOVA for repeated measures is the fundamental model for ICC. Copyright © 2016 Wolters Kluwer • All Rights Reserved 10

ICC Models for Fully Crossed Designs (cont’d) • Three factors need to be considered in selecting an ICC formula for reliability assessment: • Will a single score or an averaged score for each person be used? • Are the k measurements viewed as a fixed or random effect? • Is the assessment most concerned about consistency of scores (e. g. , ranking between observers are consistent) or absolute agreement of scores (scores across measures are identical)? • When systematic variation across observers or waves is considered relevant, then ICC for agreement should be used. • In clinical situations, absolute agreement may be more important than consistency. Copyright © 2016 Wolters Kluwer • All Rights Reserved 11

ICC Models for Not Fully Crossed Designs • Designs that are not fully crossed include: • Nested designs: Different pairs of raters rate different patients with no overlap. • Unbalanced designs: Neither crossed or nested; two raters per patient, but raters are not paired, and there is some (but not complete) overlap of patients and raters. Copyright © 2016 Wolters Kluwer • All Rights Reserved 12

Intraclass Correlation Coefficient Calculation in SPSS Copyright © 2016 Wolters Kluwer • All Rights

Intraclass Correlation Coefficient Calculation in SPSS (cont’d) Copyright © 2016 Wolters Kluwer • All

QUESTION Select the statement that is false for Intraclass Correlation Coefficient (ICC) Models and fully crossed designs: A. The two-way ANOVA for repeated measures is the fundamental model used to analyze a fully crossed design. B. Nested or unbalanced designs are two designs that are fully crossed designs. C. One type of score variability in fully crossed designs is the variation from person to person being measured for N people. D. A factor that needs to be considered when selecting an appropriate ICC formula for a fully crossed design is whether a single or an averaged score is being used. Copyright © 2016 Wolters Kluwer • All Rights Reserved 15

ANSWER Answer: B Nested or unbalanced designs are designs that are used when different pairs of raters rate different patients with no overlap (nested), or raters are not paired, and there is some overlap of patients and raters (unbalanced). Copyright © 2016 Wolters Kluwer • All Rights Reserved 16

Interpretation of ICC Values • Reliability needs to be higher for measures that will be used to make decisions about individual people. • Acceptable minimum reliability criteria for decision making about individuals range from. 85 -. 95. • Recommended minimum reliability values for measures used in group situations range from. 70 to. 75. • It should be remembered that an ICC of. 70 means that approximately 70% of the variance is attributed to the “true score, ” while approximately 30% is attributed to error. • Low ICC values may result from: • Low variability in the N by k data matrix. • Problems with the measurement design. • People-by-rater interactions. • The measure not being reliable. Copyright © 2016 Wolters Kluwer • All Rights Reserved 17

Consequences of Low ICC Values • Sample size needed to achieve a given power is greater because it is related to the value of the ICC. • There is difficulty with determining true changes after treatment. • There are misclassifications of clinical conditions and potential treatment errors. Copyright © 2016 Wolters Kluwer • All Rights Reserved 18

Reliability Parameters for Noncontinuous Variables: Proportion of Agreement • Proportion of overall agreement: total percentage of agreement on both positive and negative cases. • Proportion of negative agreement: percentage of agreement on negative (trait not present) cases. • Proportion of positive agreement: percentage of agreement on positive (trait present) cases. • Examination of specific agreement (positive and negative) is particularly important if the distribution is severely skewed. Copyright © 2016 Wolters Kluwer • All Rights Reserved 19

Kappa for Dichotomous Ratings by Two Raters • The kappa statistic is used to correct for raters’ agreement that occur by chance. • Is the most widely used reliability index. • Assumptions for use include: • People are being rated independent of one another. • All ratings should be made by the same k raters. • Rating categories are independent of one another. • Marginal homogeneity, which refers to whether the raters distribute their ratings in a comparable fashion, can be used to further understand rater agreement. Copyright © 2016 Wolters Kluwer • All Rights Reserved 20

Weighted Kappa for Ordinal Ratings • Weighted kappa: a method in which “partial credit” is given to raters whose ratings are not identical but are in close proximity to each other. • Weighting schemes include: • Linear weights. • Quadratic weights (scheme most widely used). Note: Cohen’s kappa is not appropriate for use in not fully crossed designs. Copyright © 2016 Wolters Kluwer • All Rights Reserved 21

Interpretation of Kappa • Kappa values can range from -1. 0 to +1. 0. • A kappa of 1. 0 means that all ratings are along the diagonal of the contingency table. • Although guidelines tend to be arbitrary, the following is suggested: <. 20 Poor . 21 -. 40 Fair. 61 -. 80 Substantial. 41 -. 60 Moderate >. 81 Excellent • Values under. 60 may indicate need for modifications in the instrument, the raters, the training protocol, or other aspects of the measurement situation. • Kappa paradox: occurs in skewed distributions when the proportion of agreement is substantial but kappa is low. Copyright © 2016 Wolters Kluwer • All Rights Reserved 22

QUESTION The following Cohen’s kappa (k) values strongly suggest that the instrument, the raters, the training protocol, or other aspects of the measurement situation need to be modified or there is an error in the kappa calculation (select all that apply): A. B. C. D. k k = = . 69. 20 3. 2. 80 Copyright © 2016 Wolters Kluwer • All Rights Reserved 23

ANSWER Answer: A and D In general, kappa values under. 60 may indicate need for modifications in the instrument, the raters, the training protocol, or other aspects of the measurement situation. A value of 3. 2 is not within the range of an accurately calculated kappa score. Copyright © 2016 Wolters Kluwer • All Rights Reserved 24

Reliability and Item Response Theory (IRT) • In IRT, the concept of information is usually used in lieu of reliability. • Information: a conditional expression of measurement precision for a single person that is population independent. • Test–retest reliability can be assessed with both static measures developed using IRT methods and also with computerized adaptive tests. • IRT scaling methods also have been used with items that require observational ratings, thus requiring interrater reliability assessment. Copyright © 2016 Wolters Kluwer • All Rights Reserved 25

Designing a Reliability Study Design • Demonstrating test-retest reliability usually is a case of testing the sample at two time periods. • The most straightforward interrater analyses for ICCs and kappas are with fully crossed designs; the nested design is the next best choice. Timing of Measurements • Simultaneity is possible in demonstrating interrater reliability, however, it is not possible in determining testretest, intrarater, or parallel test reliability. • Measurements timings need to be scheduled such that the likelihood that extraneous and transient measurement errors will be minimized. • Many experts advise that the interval for PRO measurements should be 1 to 2 weeks. • Physical measurements probably should be retested at shorter intervals. Copyright © 2016 Wolters Kluwer • All Rights Reserved 26

Other Design Issues in Reliability Studies • The following issues are especially important to consider in designing reliability studies: • Blinding. • Comparable measurement circumstances. • Training. • Attrition. • Random ordering of items or subscales. • Specification of an a priori standard. Copyright © 2016 Wolters Kluwer • All Rights Reserved 27

Sampling in Reliability Studies • People being measured should be representative of the population for whom the measure is designed, and raters should also be representative of the population of potential raters using the measure. • A heterogeneous sample from the population of interest should be used. • A sample size of 50 is deemed adequate in most reliability studies, however sample size can be estimated using the confidence interval around an estimated reliability coefficient. • Sample sizes for kappa are difficult to estimate because information is needed for: • the expected kappa value • the expected proportion of positive ratings Copyright © 2016 Wolters Kluwer • All Rights Reserved 28

Reporting a Reliability Study As much detail about the study as possible needs to be reported, including: • Type of reliability assessed. • Nature of the measure and its possible application. • Target population. • Sample details, including recruitment and heterogeneity. • Sample size and attrition. • Rater characteristics (if appropriate) and training. • Measurement procedures. • Data preparation. • Statistical decisions. • Statistical results. • Interpretation of results. Copyright © 2016 Wolters Kluwer • All Rights Reserved 29

QUESTION Is the following statement True or False? Choices to consider when designing reliability studies include use of blinding, possibility of subject attrition, whether measurement items should be randomly ordered, and projecting the minimum ICC or kappa values. Copyright © 2016 Wolters Kluwer • All Rights Reserved 30