Characteristics of Successful Assessment Measures Reliable Valid Efficient

Characteristics of Successful Assessment Measures • Reliable • Valid • Efficient - Time - Money - Resources • Don’t result in complaints

What Do We Mean by Reliability? • The extent to which a score from a test is consistent and free from errors of measurement

Methods of Determining Reliability • Test-retest (temporal stability) • Alternate forms (form stability) • Internal reliability (item stability) • Interrater Agreement

Test-Retest Reliability • Measures Temporal Stability Stable measures Measures expected to vary • Administration Same participants Same test Two testing periods

Test-Retest Reliability Scoring • To obtain the reliability of an instrument, the scores at time one are correlated with the scores at time two • The higher the correlation the more reliable the test

Test-Retest Reliability Problems • Sources of measurement errors: - Characteristic or attribute being measured may change over time. - Reactivity - Carry over effects • Practical problems: - Time consuming - Expensive - Inappropriate for some types of test

Standard Error of Measurement • Provides a range of estimated accuracy 1 SE = 68% confident 1. 98 SE = 95% confident • The higher the reliability of a test, the lower the standard error of measurement • Formula

Example Mean = 70, SD = 10 Reliability 68% Confidence 95% Confidence . 95 2. 24 4. 39 . 90 3. 16 6. 19 . 80 4. 47 8. 85 . 70 5. 47 10. 73 . 50 7. 07 13. 86

Practice Exercise Reliability SD . 92 4. 55 . 79 2. 21 . 85 7. 88 . 54 7. 88 . 72 4. 55 68% Confidence 95% Confidence

Exercise Answers Reliability SD 68% Confidence 95% Confidence . 92 4. 55 1. 29 2. 55 . 79 2. 21 1. 01 1. 98 . 85 7. 88 3. 05 6. 04 . 54 7. 88 5. 34 10. 57 . 72 4. 55 2. 41 4. 77

Serial Killer IQ Exercise Mean = 100, SD = 15, Reliability=. 90 IQ of 70 for death penalty Killer IQ Ed Kemper 145 Henry Lee Lucas 89 Hubert Geralds 73 Louis Craine 69 Clarence Victor 65 95% Confidence

Serial Killer IQ - Answers Mean = 100, SD = 15, Reliability=. 90 IQ of 70 for death penalty Killer IQ 95% Confidence Ed Kemper 145 135. 6 – 154. 4 Henry Lee Lucas 89 79. 6 – 98. 4 Hubert Geralds 73 63. 6 – 82. 4 Louis Craine 69 59. 6 – 78. 4 Clarence Victor 65 55. 6 – 74. 4

Alternate Forms Reliability • Establishes form stability • Used when there are two or more forms of the same test - Different questions - Same questions, but different order - Different administration or response method (e. g. , computer, oral) • Why have alternate forms? - Prevent cheating - Prevent carry over from people who take a test more than once • GRE or SAT • Promotion exams • Employment tests

Alternate Forms Reliability Administration • Two forms of the same test are developed, and to the highest degree possible, are equivalent in terms of content, response process, and statistical characteristics • One form is administered to examinees, and at some later date, the same examinees take the second form

Alternate Forms Reliability Counterbalancing Subjects First Administration Second Administration 1 -50 Form A Form B 51 -100 Form B Form A

Alternate Forms Reliability Scoring • Scores from the first form of test are correlated with scores from the second form • If the scores are highly correlated, the test has form stability

Difference Between Parallel and Equivalent Client Malone Crane Peterson Form A 6 9 11 Form B 8 11 13 Clavin Boyd Chambers Tortelli Howe Pantusso Sternin Average Score 12 12 17 18 19 21 24 14. 9 14 14 19 20 21 23 26 16. 9

Alternate Forms Reliability Disadvantages • Difficult to develop • Content sampling errors • Time sampling errors

What the Research Shows • Computer vs. Paper-Pencil - Few test score differences - Cognitive ability scores are lower on the computer for speed tests but not power tests • Item order - Few differences • Video vs. Paper-Pencil -Little difference in scores -Video reduces adverse impact

Internal Reliability • Defines measurement error strictly in terms of consistency or inconsistency in the content of the test • With this form of reliability the test is administered only once and measures item stability

Determining Internal Reliability Split-Half Method • Test items are divided into two equal parts • Scores for the two parts are correlated to get a measure of internal reliability • Need to adjust for smaller number of items • Spearman-Brown prophecy formula: (2 x split half reliability) ÷ (1 + split-half reliability)

Spearman-Brown Formula (2 x split-half correlation) (1 + split-half correlation) If we have a split-half correlation of. 60, the corrected reliability would be: (2 *. 60) ÷ (1 +. 60) = 1. 2 ÷ 1. 6 =. 75

Spearman-Brown Formula Estimating the Reliability of a Longer Test L = the number of time longer the new test will be

Example Suppose you have a test with 20 items and it has a reliability of. 50. You wonder if using a 60 -item test would result in acceptable reliability. = = Estimated New Reliability =. 75 =

Practice Split-Half # of # Reliability items Proposed items. 80. 40. 80 30 45 . 40 10 50 Corrected Future Reliability

Practice Answers Split-Half # of # Corrected Future Reliability items Proposed Reliability items. 80. 89. 40 . 57 . 80 30 45 . 86 . 40 10 50 . 77

Common Methods to Determine Internal Reliability • Cronbach’s Coefficient Alpha - Used with ratio or interval data. • Kuder-Richardson Formula - Used for test with dichotomous items - yes-no - true-false - right-wrong

Interrater Reliability • Used when human judgment of performance is involved in the selection process • Refers to the degree of agreement between 2 or more raters • 3 common methods used to determine interrater reliability - Percent agreement - Correlation - Cohen’s Kappa

Interrater Reliability Methods Percent Agreement • Determined by dividing the total number of agreements by the total number of observations • Problems - Exact match? - Very high or very low frequency behaviors can inflate agreement

Interrater Reliability Methods Correlation • Ratings of two judges are correlated • Pearson for interval or ratio data and Spearman for ordinal data (ranks) • Problems - Shows pattern similarity but not similarity of actual ratings

Interrater Reliability Methods Cohen’s Kappa • Allows one to determine not only the level of agreement, but the level that would be determined by chance • A Kappa of. 70 or higher is considered acceptable agreement

Forensic Examiner A Insane Malingering 16 0 16 1 3 4 17 3 20 Forensic Examiner B Malingering

Example

Demonstration Television Show Elimidate Law and Order Friends CSI X-Files Walker Texas Ranger Just Shoot Me Will and Grace Darma and Greg JAG Like (1) or Dislike (0)

Increasing Rater Reliability • Have clear guidelines regarding various levels of performance • Train raters • Practice rating and provide feedback

Scorer Reliability • Allard, Butler, Faust, & Shea (1995) - 53% of hand scored personality tests contained at least one error - 19% contained enough errors to alter a clinical diagnosis

The degree to which inferences from scores on tests or assessments are justified by the evidence

Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. . The process of validation involves accumulating evidence to provide a sound scientific basis for the proposed score interpretations. It is the interpretations of test scores required by proposed uses that are evaluated, not the test itself. When test scores are used or interpreted in more than one way, each intended interpretation must be validated. Sources of validity evidence include but not limited to: evidence based on test content, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, evidence based on consequences of testing. Standards for Educational and Psychological Testing (1999)

Common Methods of Determining Validity • • • Content Validity Criterion Validity Construct Validity Known Group Validity Face Validity

Content Validity • The extent to which test items sample the content that they are supposed to measure • In industry the appropriate content of a test of test battery is determined by a job analysis • Considerations - The content that is actually in the test - The content that is not in the test - The knowledge and skill needed to answer the question

Test of Logic • • Stag is to deer as ___ is to human Butch is to Sundance as ___ is to Sinatra Porche is to cars as Gucci is to ____ Puck is to hockey as ___ is to soccer What is the content of this exam?

Messick (1995) Sources of Invalidity • Construct underrepresentation • Construct-irrelevant variance - Construct-irrelevant difficulty - Construct-irrelevant easiness

Domain Content Test Content

Criterion Validity • • Criterion validity refers to the extent to which a test score is related to some measure of job performance called a criterion Established using one of the following research designs: - Concurrent Validity - Predictive Validity - Validity Generalization

Concurrent Validity • Uses current employees • Range restriction can be a problem

Predictive Validity • Correlates test scores with future behavior • Reduces the problem of range restriction • May not be practical

Validity Generalization • Validity Generalization is the extent to which a test found valid for a job in one location is valid for the same job in a different location • The key to establishing validity generalization is meta-analysis and job analysis

Construct Validity • The extent to which a test actually measures the construct that it purports to measure • Is concerned with inferences about test scores • Determined by correlating scores on a test with scores from other test

Construct Valid Measures • Correlate highly with measures of similar constructs • Don’t correlated highly with irrelevant or competing constructs • Correlate with other measures of the construct that are measured in different ways • Don’t correlate highly with competing or irrelevant constructs that are measured in similar ways

Messick (1995) Facets of Validity Evidential Basis Consequential Basis Test Interpretation Test Use Construct Validity Relevance & Utility Value Implications Social Consequence

Face Validity • • • The extent to which a test appears to be job related Reduces the chance of legal challenge Increases test taker motivation Increases acceptance of test results Increasing face validity - Item content - Professional look of the test - Explaining the nature of the test

Known-Group Validity • Compares test scores of groups “known” to be different • If no differences, test may not be valid • If differences, conclusions are hard to make