Reliability Summary True Score Variance Reliability Reliability Summary
Reliability: Summary True Score Variance & Reliability:
Reliability: Summary Example: Reliability of a single test with a single latent construct: Reliability
Reliability: Summary Reliability of a single test with multiple latent constructs: Matrix computations The term: represents the true score variance:
Reliability: Summary Reliability of a single test with multiple latent constructs: Matrix computations We thus have: q True score variance: q Variance of Y: q Reliability of Y:
Reliability: Summary General comments concerning reliability: 1. The concept of reliability refers to the test and not to the construct, i. e. , a test can be (un-) reliable but not a construct. 2. Reliability is a population dependent concept: It can be estimated only if construct values vary (within a population). Otherwise the reliability is 0. 3. Reliability is a theoretical construct that cannot be observed directly. It can be measured only in a model dependent way, i. e. , assumptions about the measurement model are required and the measured reliability depends on the correctness of these assumptions.
Reliability: Summary q Traditional approaches to measure reliability: q q q Test-Retest Method: Same test applied twice. Parallel forms: Two equivalent versions of the test. Test halves: Test is split in 2 halves (e. g. even vs. uneven). q The traditional approaches are based on the assumption that test are parallel. q Traditional approaches conceal the fact that reliability can be measured only in a model dependent way.
Chapter 4: Reliability: Concept and estimation o
Chapter 4: Reliability: Concept and estimation o
Chapter 4: Reliability: Concept and estimation On the limits of traditional measures of the reliability of sum scores: 1. The traditional measures of test scores (Spearman. Brown, coefficient and Guttman’s 2 are unbiased only in case of certain assumptions to be met: q parallel tests in case of Spearman-Brown. q -equivalent tests in case of and 2. 2. For congeneric measures or measures loading on more than one latent construct, and without correlated errors and 2 underestimate the true reliability. 3. 2 underestimates the reliability less than .
Chapter 4: Reliability: Concept and estimation On the limits of traditional measures of the reliability of sum scores: 4. In case of correlated errors the three traditional estimates of the reliability of sums can overestimate the reliability of the sum (An example is given below and in the exercises). 5. The traditional measures use observed correlations and (co-) variances and not the model implied correlations and (co-) variances. The latter are to be preferred, in case of the model being approximately correct, since they are more robust. 6. If an analysis of the covariance structure of the tests has been performed an unbiased estimate can be computed.
Chapter 4: Reliability: Concept and estimation Method 4 -4: Determining the reliability of the weighted sum of test scores in the general factor analytic model: True score variance of the weighted sum: Total variance of the weighted sum: Reliability:
Chapter 4: Reliability: Concept and estimation o
Chapter 4: Reliability: Concept and estimation Method 4 -4: Determining the reliability of the weighted sum of test scores in the general factor analytic model Summary: The relevant matrices: q q
Chapter 4: Reliability: Concept and estimation Over- and underestimation of true reliability by coefficient , Guttmans 2, and the Spearman. Brown coefficient: 1. and 2 are unbiased estimates of the sum of tests in case of -equivalent tests. The Spearman-Brown is unbiased in case of parallel tests. 2. and 2 underestimate the true reliability in case of: a) Congeneric measures. b) Measures are loading on more than a single latent construct and errors are uncorrelated.
Chapter 4: Reliability: Concept and estimation Over- and underestimation of true reliability by coefficient , Guttmans 2, and the Spearman. Brown coefficient: 3. In case of correlated errors, and 2 can overestimate the true reliablity of the sum of tests.
Chapter 4: Reliability: Concept and estimation Overestimation of true reliability by coefficient and Guttmans 2:
Chapter 4: Reliability: Concept and estimation Overestimation of the true reliability by the Spearman-Brown coefficient: (Exercise 4 -18)
Chapter 4: Reliability: Concept and estimation q A faulty interpretation of coefficient : q must not be interpreted as an index of homogeneity, i. e. that the tests are measuring a single underlying construct. q can be high despite the fact that there are many underlying constructs (cf. the next example). q To test for homogeneity, fit the congeneric model.
Chapter 4: Reliability: Concept and estimation Requirements on monotony of the reliability of the sum of tests: 1. Addition of a reliable test to an existing set of tests should increase the reliability of the sum. 2. Replacing a test by a more reliable one should increase the reliability of the sum. 3. Reduction of the correlation between two latent constructs should result in a decrease of the reliability of the sum.
Chapter 4: Reliability: Concept and estimation q Summary: Problems associated with the reliability of unweighted sums: q In general, unweighted sums do not result in maximal reliabilities. q Unweighted sums can lead to violations of monotonicity requirements. q Maximal reliability can be obtained by maximizing the reliability of the weighted test items: with respect to the weights w:
- Slides: 20
