Chapter 4 Predictors Psychological Assessments Psych ology Appli

Chapter 4 Predictors: Psychological Assessments Psych ology Appli ed to Work ® 1

Learning Objectives Identify the major types of reliability and what they measure. Understand the major manifestations of validity and what they measure. Know the major types of psychological tests categorized by content. Explain the role of psychological testing in making assessments of people, including ethical issues and predictive accuracy. Explain non-test predictors such as interviews, assessment centers, physical ability testing, work samples, biographical information, and letters of recommendation. Understand the controversial methods of assessment. 2 Psychology Applied to Work®

Overview Predictor: Any variable used to forecast a criterion Two psychometric criteria: reliability and validity See Hogan’s Guide to Reliability and Validity 3 Psychology Applied to Work®

Reliability Consistency and stability of measurement Four types: 1. Test-retest Reliability: Coefficient of stability 2. Equivalent-Form Reliability: Coefficient of equivalence 3. Internal-Consistency Reliability: Homogeneous content 4. Split-half Reliability Cronbach’s Alpha or Kuder-Richardson 20 (KR 20) Inter-Rater Reliability: Conspect reliability 4 Psychology Applied to Work®

Validity Accuracy of measurement Construct Validity – Single unitary concept Continuous process of verifying the accuracy of inferences among concepts Convergent Divergent validity 5 Psychology Applied to Work®

6 Psychology Applied to Work®

Validity (cont’d) Criterion-Related Validity Concurrent criterion-related validity Predictive criterion-related validity Validity Coefficient 7 Psychology Applied to Work®

Validity (cont’d) Content Validity Representative sample of subject domain Subject Matter Experts Similar type of "validity" - Face Validity Items appear appropriate for purpose of assessment 8 Psychology Applied to Work®

Predictor Development Measure construct directly (e. g. mechanical comprehension) sample of behavior (demonstrate on the job) “Fix it” Measure current - Job interview – e. g. interpersonal skills past behavior – letter of recommendation (could also be done in an interview, e. g. “give me an example of when you…. ”) 9 Psychology Applied to Work®

The Purpose of Testing Tests have right/wrong answers, surveys don't But both are considered “tests” by EEOC Low-stakes vs. moderate-stakes v. high-stakes testing Give an example of each 10 Psychology Applied to Work®

Test Content Intelligence Concept of g How do you define “g”? Single best predictor job performance Multiple intellectual abilities What are some? What do Air Traffic Controllers need? Academic vs. practical vs. creative intelligence Mechanical Aptitude Bennett-Mechanical Test 11 Psychology Applied to Work®

Test Content (cont’d) Personality No “right v. wrong” answers Scale scores used to predict job success Big 5 Personality Theory Dark Triad EEOAC Machiavellianism, Narcissism, psychopathy Concerns with faking What can be done about it? Which is a better measure of personality? Self or observer? (Oh et al. 2011) 12 Psychology Applied to Work®

Test Content (cont’d) Integrity Overt (transparent) Personality-based More predictive of counterproductive work behavior than job performance or turnover (Van Iddekinge et al. ‘ 13) Why do you think it is so difficult to validate? 13 Psychology Applied to Work®

Test Content (cont’d) Situational Judgment (SJT) Responses are not scored simply as right or wrong So how are they scored? Designed to reflect reality of making decisions in life Think of one for your project job Computerized Adaptive Testing (CAT) Automated, difficulty level of questions pre-calibrated Used by the military and other large volume assessments (SAT, GRE) 14 Psychology Applied to Work®

Test Content (cont’d) Testing on the Internet The move from paper-and-pencil testing to online computer testing (“paper to pixels”) What are some advantages and disadvantages? Proctored v. unproctored web-based testing Would you be more likely to want to use a cognitive ability test or personality inventory online, unproctored? 15 Psychology Applied to Work®

Testing in Retrospect Trends: Tyranny of testing: critical decisions based on a single test Inducements to cheat on employment tests Some tests are useful in predicting job performance, others are not As a class, moderately predictive; but could be due to poor criterion specification 16 Psychology Applied to Work®

Interviews Value Universal in its appeal Criterion-related validity Serves other personnel functions achieved through its use Degree of Structure More structure increases validity and fairness Situational Interviews Situational (hypothetical) question 17 Psychology Applied to Work®

Work Samples High fidelity (replica) simulations Effective in blue-collar jobs Effective at predicting “can do, ” but not potential 18 Psychology Applied to Work®

Situational Exercises White collar counterpart of work samples; In-basket test Leaderless group discussion Modest validity (could be due to poor criterion definition) and very costly Low fidelity but could still be a reasonable compromise 19 Psychology Applied to Work®

Assessment Centers Five Characteristics 1. Managers intended candidates – selection, promotion, training 2. Assessed in group context 3. Performance is rated by assessors 4. Numerous performance dimensions assessed 5. Variety of group exercises and inventories over 1 -3 days Effectiveness Select talented people (AC ratings correlate with advancement) May be predictive because of common stereotypes held 20 Psychology Applied to Work®

Physical Abilities Testing Three main constructs: 1. Strength 2. Explosive strength 3. Gross body coordination 4. Stamina (Endurance) 1. Why is it difficult to assess in a work sample? 21 Psychology Applied to Work®

Biographical Information Past life experiences used to predict future behavior F. Mael says to ask about past behavior, not intentions – Why? Usefulness Typically high validity Reveals consistent patterns of behavior in our lives Often locates unique criterion variance Legally defensible, but problematic Problems Fairness issues Invasion of privacy Fakable 22 Psychology Applied to Work®

Letters of Recommendation Commonly used selection method but least valid: restricted range Primary value if negative information about candidate is revealed 23 Psychology Applied to Work®

Drug Testing Substance abuse is major global problem Two types of assessments Screening test Confirmation test Problems of reliability Practical issues Cost savings Issues of uniform drug testing Some issues outside the scope of I/O psychology 24 Psychology Applied to Work®

Controversial Methods of Assessment Polygraphy or Lie Detection President Reagan 1988 ban on private-sector use Increasing use in government and security agencies Test of Emotional Intelligence Highly controversial and still not well-established 25 Psychology Applied to Work®

Overview and Evaluation of Predictors Four major evaluative standards 1. Validity: Predictive accuracy 2. Fairness: Differential predictive accuracy across different groups 3. Applicability: Interview highest applicability and most universally used 4. Cost: Subjective and arbitrary, but highly relevant to organizations 26 Psychology Applied to Work®

Overview and Evaluation of Predictors 27 Psychology Applied to Work®

Overview and Evaluation of Predictors rarely meet all standards in practice Three dimensions to judgments of job-related experience 1. Personal attributes 2. Perceived outcome of experience 3. Aspects of experience judged relevance and important 28 Psychology Applied to Work®