Chapter 4 Classical Test Theory CTT Overview q

Chapter 4: Classical Test Theory (CTT): Overview q Preliminaries: Covariance matrix q Exposition of theory Ø What is modeled by CTT? Ø Decomposition of test scores and axioms of CTT Ø Classical test models of CTT q Structural equation models (SEM) Ø Exposition of the basic assumptions Ø Implementation of the axioms using SEM Ø Reconstruction of test models of CTT using SEM q Reliability, Validity, Latent Scores, Groups

Chapter 4: Classical Test Theory (CTT): Preliminaries I q Structure of a covariance matrix: Ø Given: Two sets of variables: and Ø The covariance matrix of the two sets of variables is given by: Ø 5 types of entries: 2 types of variances: , 3 types of covariances: within first group within second group between groups

Chapter 4: Classical Test Theory (CTT): Preliminaries II q Structure of a covariance matrix: Ø The covariance matrix can thus be partitioned into 4 parts:

Chapter 4: Classical Test Theory (CTT): Preliminaries III q Structure of a covariance matrix: Ø Partitioning of the covariance matrix: Ø Variance covariance matrix of the first group of variables: Ø Variance covariance matrix of the second group of variables: Ø Matrix of the covariances between the members from different groups of variables: (the symbol T indicates transposition of the matrix)

Chapter 4: Classical Test Theory (CTT): Preliminaries IV q Covariance matrices: Some basic results Given: are n random variables, and are m random variables. The following basic facts hold (cf. Exercise 4 1): 1. The variance Var( ) of the sum of the variables corresponds to the sum of the entries of the covariance matrix and the variance Var( ) of the sum corresponds to the sum of the entries of. 2. The covariance between the two sums Cov( , ) corresponds to the sum of the entries of (or ).

Chapter 4: Classical Test Theory (CTT): Preliminaries V q Covariance matrices: Some basic results Given: are n random variables The following basic facts hold: 1. The number of entries of the covariance matrix is n 2. 2. The number of variances is n (=number of diagonal elements of ). 3. The number of covariances in is 4. The number of unique covariances is: . . 5. The number of variance and unique covariances:

Chapter 4: Classical Test Theory (CTT) Exposition of theory: What is modeled in CTT? q Expectations (Mean structure) q Variances and Covariances (Covariance structure) CTT imposes restrictions on the covariance and mean structure in two ways: q Specification of axioms of CTT q Classical test models of CTT No distributional assumptions (weak true score theory vs. strong true score theory)

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Decomposition of test scores: q At the heart of CTT lies the decomposition of the test scores into true score and error score: Ypi represents the test score of person p on test i pi represents the true score of person p on test i pi represents the error score of person p on test i q Y and are random variables (thus they are italicized), is a parameter (a fixed value for person p and test i).

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Decomposition of test scores: q It is further assumed that the expectation of the errors is zero: q By consequence (due to the rule for computing expectations), Thus, the true score is the expected value of the test scores with repeated application of the same test to the same person (the expectations are taken with respect to a within subject distribution, see below).

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Decomposition of test scores: q Problem: How to estimate the true score and the variance of the error variable? q Repeated application of the same test to the same person would result in an empirical distribution of test scores. The mean of the observed test scores may be used as an estimate of true score , whereas the variance of test scores may serve to estimate the variance of the error . q The possibility of repeated application is not feasible in practice. q CTT takes a different approach.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: Concept: Axiom: An axiom is a statement that is assumed to be true. q The three axioms of CTT are: q The axioms represent claims about the covariance structure.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: q The first axioms of CTT, states that true scores and errors of the same test i are uncorrelated within the population of examinees. The distribution of test persons is relevant and not the within subject distribution used for defining the true score. q The second axiom of CTT, implements the assumption that true score and error of different tests, i and j, are uncorrelated.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: q The first and second axiom could by integrated into a single axiom that states that true and error scores are in general uncorrelated. q The third axioms of CTT, implements the constraint of uncorrelated errors. Comment: The variance and covariance statements of the axioms refer to variations within the population of examinees (between subject variation) and not to within subject variations.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: Example: Simplification of the covariance structure afforded by the axioms of CTT: Given: q 3 random variables Y 1, Y 2, and Y 3 representing observed scores. q 3 random variables of the three tests. 1, 2, and 3 representing true scores q 3 random variables 1, 2, and 3 representing errors.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: Example: Simplification of the covariance structure afforded by the axioms of CTT: The covariance matrix representing the covariance structure of true scores and errors:

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: Example: Simplification of the covariance structure afforded by the axioms of CTT: Application of the axioms results in the following simpler covariance matrix : upper right lower left lower right off diagonal

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: Example: Simplification of the covariance structure afforded by the axioms of CTT: The covariance matrix Y of observed scores can be computed using covariance algebra, specifically the simple relationship. This results in the following covariance matrix of observed scores:

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: Example: Simplification of the covariance structure afforded by the axioms of CTT: The covariance equations represent variances and covariances of observed scores in terms of variances and covariances of latent and error scores: They can be recovered from the covariance matrix of observed scores:

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Axioms of CTT: Example: Simplification of the covariance structure afforded by the axioms of CTT: Model parameters are identified, if they can be determined uniquely from the covariance equations. q The covariance parameters of the latent constructs are identified. q The variance parameters are not identified. q Thus the axioms of CTT do not specify enough restriction to identify the vari ance parameters. q The parameters cannot be estimated uniquely.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: q Thus the axioms of CTT do not specify enough restriction to identify the parameters of the general test model of CTT. q Consequently, additional restrictions are required. This is accomplished by the classical test models q 3 classical test models: 1. The congeneric model 2. The equivalent model 3. The parallel model.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: q 3 classical test models: 1. The congeneric model 2. The equivalent model 3. The parallel model. q The equivalent model is a special case of the congeneric model (i. e. it results from the congeneric model by adding restrictions. q Similarly, the parallel model is special case of the equivalent model. q The three models are nested (embedded).

Kapitel 2: Klassische Testtheorie Exposition of theory: Classical test models: For the following discussion of the three classical test model we use the following notation: denote the observed scores for the n tests. denote the true scores for the n tests. denote the measurement errors for the n tests.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: Congeneric test model Choose one of the tests, e. g. the first test, as a reference. The true scores of the other tests are linear functions of the true score of the reference test, in symbols: The congeneric model entails a restriction of the covariance struc ture of the truescores.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: Congeneric test model Covariance of true scores implied by the congeneric model: Variances: Covariances:

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Example: Covariance matrix of true scores of four tests implied by the congeneric model: Important: The true scores are perfectly correlated.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Example: Covariance matrix of true scores of four tests implied by the congeneric model: The true scores are perfectly correlated. This follows from the definition of correlations:

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Example: Simplification of the covariance structure achieved by the congeneric model: unique variances and covariances (of true scores) in case of 4 tests. only 4 free parameters of the model to represent these variances and covariances: , , , and

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Example: Model implied covariance matrix of the observed scores: The model comprises 8 parameters for modeling the 10 observed (co ) variances:

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Example: Model parameters and observed scores: The 8 model parameters: can be determined from the covariance equations (also called model equations) that follow directly from the model implied covariance matrix of the observed scores. The model equations link observed (co ) variances to the model implied ones.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Congeneric test model (Summary): q The linear relationship between true scores results in simplification of the covariance structure of true scores. Specifically the correlations between true scores are assumed to be 1. 0. q The number of free parameter increases less strongly than the number of observed variances and covariances. By consequence: Ø With 3 test the model is exactly identified. Ø With 4 and more tests the model can be tested statistically.

Chapter 4: Classical Test Theory (CTT) o

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: (Essential) -equivalent test model Implied covariance structure of true scores and of observed scores: The model makes the (strong) prediction that the observed covariances between all test items are equal.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: (Essential) -equivalent test model Estimation and testability of the model: q The parameters of the essential equivalent model can be estimated with two tests only. q The model can be tested with three tests: The model predicts that the covariances between the three pairs of tests are identical. q Adding a new test to n existing ones adds n+1 ob served data (n covariances and 1 variance). The numbers of free parameters increases only by 1 (= the variance of the error term).

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: (Strictly) parallel test model The parallel test model is a special case of the essential equivalent model that results from the latter by assuming (in addition) that the error variances are all equal: The strict parallel model assumes the same additional restriction (on the mean structure) as the equivalent model that all means are the same: .

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: Parallel test model Implied covariance structure of true scores and of observed scores: The model makes the predictions that the observed covariances between all test items are equal, and that the variances of the test scores are also the same.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: Parallel test model: Estimation and testability of the model: q The parameters of the parallel model can be esti mated with two tests since the model comprises 2 parameters only: and. q The model can be tested with two tests only: In this case, the model predicts that the variances of the two tests are the same (in addition it predicts that the covariances and correlations between tests are the same). q Adding a new test to n existing ones adds n+1 ob served data (n covariances and 1 variance). No additional free parameter is required.

Chapter 4: Classical Test Theory (CTT) Exposition of theory: Classical test models: Concept: Parallel test model Estimation of true score and error variance q Strictly parallel test are perfectly equivalent. q Thus, instead of repeated application of the same test parallel tests can be applied to the same examinee. q The mean of the test scores from the different tests provides an estimate of the examinee’s true score. q The error variance of the observed test scores of the examinee can be used as an estimate of the error variance.

Chapter 4: Classical Test Theory (CTT) q Criticism of CTT: Two problematic features 1. The conception of true scores as expectations 2. The assumption of a within subject distribution q True score conception: The true score is the mean of test scores with repeated testing. Lord & Novick provide a thought experiment: «Suppose we ask an individual, Mr. Brown, repeatedly whether he is in favor of the United Nations; suppose further that after each question we “wash his brains” and ask him the same question again. Because Mr. Brown is not certain as to how he feels about the United Nations, he will sometimes give a favorable and sometimes an unfavorable answer. Having gone through this procedure many times, we then compute the proportion of times Mr. Brown was in favor of the United Nations…» (Lord & Novick, 1968, page 29).

Chapter 4: Classical Test Theory (CTT) q True score vs. latent variable conception: Ø For each person and test there exists a true score. The true score is thus test dependent and does not exist independently of a test. Ø Latent variable conception: The latent ability exists independently of the test. The latter is just an instru ment for measuring the latent ability. q Problems of the true score conception: 1. Inflation of true scores. 2. True score conception does not guarantee that the test measures something useful. 3. Problem of relationship between true scores and latent abilities.

Chapter 4: Classical Test Theory (CTT) q Possibly suitability of the true score conception: Ø The true score conception may be useful within non scientific contexts where tests are used as selection tools. Ø For example, LSAT as a test for entrance to law schools. Ø Empirical validity, that is, the correlation between test scores and study success can be used for assessing the adequacy of the test.

Chapter 4: Classical Test Theory (CTT) q Problematic aspects of the within subjects distri bution: Ø Only required for the definition of the true score. Ø Not required for specification of the axioms of CTT. Ø Not required for the concept of reliability (that is based on the population of true scores). q Summary: The true score conception and the within subject distribution, required for its definition, are of little value in scientific contexts where theoretical constructs are to be measured.