Testing the Test Presenter Lisa Harpe Ph D
- Slides: 46
Testing the Test Presenter: Lisa Harpe, Ph. D. Peoplefluent Research Institute (PRI) © People. Fluent 2014
Agenda • • • Regulatory Climate What is a Test? What is Validity? Questions for a Third Party Test Vendor Questions for a Test User Documentation Checklists © 2014 People. Fluent | Confidential Presentation Title | date 2
Regulatory Climate • EEOC • Experts on staff • Charges • Some well known cases in the mid-2000 s: Dial Corp, Whirlpool, Fed-Ex • Criminal Background Checks • ADA violations • OFCCP • Experts on staff • Settlements (in last 2 years) • Leprino Foods • GE • Lincoln Electric © 2014 People. Fluent | Confidential Presentation Title | date 3
UGESP • Uniform Guidelines on Employment Selection Procedures (UGESP) (1978) • Adverse impact (A. I) • No need to show validity if no A. I. • Justify, change, find alternative or eliminate if A. I. • Justification • Validity evidence = job related • Business necessity © 2014 People. Fluent | Confidential Presentation Title | date 4
UGESP Citations • Section 60 -3, Uniform Guidelines on Employee Selection Procedure (1978); 43 FR 38295 (August 25, 1978). • Equal Employment Opportunity Commission: 29 CFR Part 1607 • Department of Labor / Office of Federal Contract Compliance Programs: 41 CFR Part 60 -3 • Department of Justice: 28 CFR 50. 14 • Civil Service Commission: 5 CFR 300. 103(c) © 2014 People. Fluent | Confidential Presentation Title | date 5
Resources • Uniform Guidelines on Employment Selection Procedures (UGESP, 1978) • Department of Labor’s (DOL) Testing and Assessment: an Employer’s Guide to Good Practices (1999) • American Psychological Association’s (APA) Standards for Educational and Psychological Testing (2014) • Society of Industrial Organizational Psychologist’s (SIOP) Principles for the Validation and Use of Personnel Selection Procedures (4 th edition, 2003) © 2014 People. Fluent | Confidential Presentation Title | date 6
What is a Test? • ‘Any instrument or procedure that samples behavior or performance’ (DOL Guide) • ‘Any device or procedure that samples behavior which is scored in a standard process’ (APA Standards) • ‘Any procedure used singly or in combination to make personnel decisions’ (‘selection procedure, test, assessment, and predictor’ = interchangeable terms) (SIOP Principles) • vs. basic qualification questionnaire © 2014 People. Fluent | Confidential Presentation Title | date 7
Types of Tests • • • Cognitive ability tests Physical ability tests Achievement tests Personality tests Job knowledge tests Skills tests Honesty and integrity tests Biodata inventories Work samples and simulations Assessment centers © 2014 People. Fluent | Confidential Presentation Title | date 8
Types of Tests • Employment interviews • Education and experience requirements (including licensing and certification) • Recommendations and reference checks • Criminal record check • Medical examinations • Drug and alcohol tests © 2014 People. Fluent | Confidential Presentation Title | date 9
What is Validity? • Principles advocates the Standards’ definition as “the degree to which accumulated evidence and theory support specific interpretations of test scores entailed by proposed uses of a test” (AERA et al. , 1999, p. 184) © 2014 People. Fluent | Confidential Presentation Title | date 10
Validity • Validity depends on the purpose of the test • Measure of extraversion vs. predictor of job success vs. predictor of training success • Selection vs. promotion vs. placement • Target population • Validity depends on the use of the test results • Cut score (Pass/Fail) • Ranking • Part of assessment battery © 2014 People. Fluent | Confidential Presentation Title | date 11
Sources of Validity Evidence • Content validity • Can you show a direct link between the test items and job behaviors, work outcomes or KSAOs? • Criterion related validity • Can you show that the test is related to performance? • Construct validity • Are there multiple lines of evidence that whatever is being measured by the test is related to the job? • Other • Response Process • Internal structure • Consequential © 2014 People. Fluent | Confidential Presentation Title | date 12
Content Validity Work behaviors, activities, outcomes or KSAOs Test © 2014 People. Fluent | Confidential Presentation Title | date 13
Content Validity • Expert Judgment • SME • • Incumbents Hiring manager HR Industrial Psychologists • Define Important Work Behaviors, Activities and Outcomes • Evaluate Test Content for Relevance © 2014 People. Fluent | Confidential Presentation Title | date 14
Content Validity Job Representative Work Test Content Behavior Administrative Assistant Maintain supervisor’s calendar Microsoft Outlook Skills Assessment Administrative Assistant Word processing Microsoft Word Skills Assessment Truck Driver Drive truck Truck Driving Test © 2014 People. Fluent | Confidential Presentation Title | date 15
Criterion Related Validity • Criterion Measures • Performance ratings • 6 month, annual • Supervisor, peer, customer / client • Productivity • Sales revenue • Turnover • Absenteeism • Multi-dimensional © 2014 People. Fluent | Confidential Presentation Title | date 16
© 2014 People. Fluent | Confidential Presentation Title | date 17
Criterion-related validity • Predictive: Can test results tell us how well a person will perform in the job? • Concurrent: Are test results related to job performance? © 2014 People. Fluent | Confidential Presentation Title | date 18
Criterion-related Validity Correlation Coefficient (r) © 2014 People. Fluent | Confidential Presentation Title | date 19
Criterion-related Validity Correlation Coefficient (r) © 2014 People. Fluent | Confidential Presentation Title | date 20
Criterion-related Validity Correlation Coefficient (r) © 2014 People. Fluent | Confidential Presentation Title | date 21
Criterion-related Validity • Question 1: Is the correlation statistically significant? • Question 2: Is the correlation practically significant? © 2014 People. Fluent | Confidential Presentation Title | date 22
Criterion-related Validity Guidelines for Interpreting Correlation Coefficients (DOL guide) Correlation coefficient value Interpretation above. 35 very beneficial . 21 -. 35 likely to be useful . 11 -. 20 depends on circumstances below. 11 unlikely to be useful © 2014 People. Fluent | Confidential Presentation Title | date 23
Criterion-related Validity Coefficient of Determination (R-squared) Test R-sq =. 75 © 2014 People. Fluent | Confidential Presentation Title | date Job Performance 24
Criterion-related Validity Coefficient of Determination (R-squared) Test R-sq =. 00 Job Performance © 2014 People. Fluent | Confidential Presentation Title | date 25
Criterion-related Validity Coefficient of Determination (R-squared) Test R-sq =. 2 Job Performance © 2014 People. Fluent | Confidential Presentation Title | date 26
© 2014 People. Fluent | Confidential Presentation Title | date 27
What Influences the Validity Coefficient • Validity of BOTH the test (predictor) and performance measure (criterion) • Reliability of BOTH the test (predictor) and performance measure (criterion) • Range restriction in the test (predictor) and / or performance measure (criterion) • Sample size affects the significance of r © 2014 People. Fluent | Confidential Presentation Title | date 28
Construct validity • Construct = unobservable trait or characteristic • Examples • • • Extraversion Customer service orientation Intelligence Oral presentation skills Ability to trouble-shoot Leadership © 2014 People. Fluent | Confidential Presentation Title | date 29
Construct Validity Predictor Measure Criterion Measure (e. g. , Call. Center. Pro) (e. g. , Supervisor Rating) Predictor Construct Domain Criterion Construct Domain (e. g. , People Skills) (e. g. , Interpersonal Interactions) © 2014 People. Fluent | Confidential Presentation Title | date 30
Other Types of Validity Evidence • Face Validity • Response Process • Internal Structure • Consequential © 2014 People. Fluent | Confidential Presentation Title | date 31
Job Analysis • Why Do I Care About This? • Analysis of Work (SIOP) • 2 Purposes • Develop predictors (tests / selection procedures) • Develop criteria (performance measures) • Approaches • • Questionnaires Observations Interviews Documentation • Currency © 2014 People. Fluent | Confidential Presentation Title | date 32
Reliability • Reliability = Consistency or Stability • Why Do I Care About This? • Types: • • Test-retest Interrater reliability Internal consistency Alternate form © 2014 People. Fluent | Confidential Presentation Title | date 33
Reliability Guidelines for Interpreting Reliability Coefficients (DOL) Reliability Coefficient Interpretation . 90 and up excellent . 80 -. 89 good . 70 -. 79 adequate below. 70 may have limited applicability © 2014 People. Fluent | Confidential Presentation Title | date 34
Validity Generalization • Meta-analysis • Synthetic Validity • Transportability • • • Job similarity Job context similarity Criterion similarity Test use similarity Test taker similarity • Test fairness © 2014 People. Fluent | Confidential Presentation Title | date 35
Third Party Tests • Sources of Validity Evidence • Vendor manuals and tech reports • Independent reviews • Research literature • Mental Measurements Yearbook (Buros Institute of Mental Measurements) • Tests in Print (TIP) (Buros Institute of Mental Measurements) © 2014 People. Fluent | Confidential Presentation Title | date 36
Questions for a Test Vendor • What are your credentials and experience? Can you provide references? • Do you have a technical report that describes the development and validation of the test? Will this report be updated over time? If so, how will you inform me of the updates? • What costs are involved with using the test in my selection process? • For which job(s) is the test valid? Can you show that the test is valid for my company and my job(s)? © 2014 People. Fluent | Confidential Presentation Title | date 37
Questions for a Test Vendor • Can you show that the test is reliable? • Can you show that the test is not biased? Can you provide the results of adverse impact studies by race and gender? Can you conduct adverse impact studies for my job(s)? • Can you provide information on test administration, scoring and interpretation? • Will you evaluate the effectiveness of the test in my company? Will you make changes if warranted? © 2014 People. Fluent | Confidential Presentation Title | date 38
What to ask a test vendor • Will the test by changed or updated? How will you inform me of any changes or updates? • Can you document the validity of the test in accordance with the UGESP requirements? • If the vendor can not provide an answer or does not know what you are talking about, run don’t walk! © 2014 People. Fluent | Confidential Presentation Title | date 39
Questions for the Test User • What employment tests are used in the selection process? • For which job(s)? • Where in the process? • Does the test have an adverse impact? • Who developed this test? where did we get the test? • Did we consider any alternatives? • Where is the job analysis for this job? © 2014 People. Fluent | Confidential Presentation Title | date 40
Questions for the Test User • How are the test results used? • Is the test reliable? • Do we have evidence of the test’s validity? Does the evidence meet the UGESP and SIOP criteria for documentation? © 2014 People. Fluent | Confidential Presentation Title | date 41
Common Issues • Homemade Tests • See recommended process for work sample tests • Tests found on the Internet • Off the shelf tests ‘valid’ for a given industry © 2014 People. Fluent | Confidential Presentation Title | date 42
RECOMMENDED PROCESS FOR WORK SAMPLE TEST • Establish a ‘test’ panel of subject matter experts (individuals who know the job and what the job requires) – 1 -2 supervisors, 2 -3 incumbents (the more the better) with more than 1 year experience, HR rep. • Is test related to job? • Identify and document what knowledge, skills or abilities (KSA) are measured by the test. • Document that KSAs measured by test are required on the job • • Provide examples of tasks and duties representing the KSAs Indicate frequency of performing tasks or duties requiring the KSAs Indicate the importance of performing tasks or duties requiring the KSAs Are the KSAs needed on the first day of the job? © 2014 People. Fluent | Confidential Presentation Title | date 43
RECOMMENDED PROCESS FOR WORK SAMPLE TEST • How to score test? • Experts define criteria as specifically as possible • Panel of SMEs should score the test • Panel should agree on definition of success • Conduct an annual review of the test • Relationship to performance (could include turnover) • Adverse impact © 2014 People. Fluent | Confidential Presentation Title | date 44
RECOMMENDED PROCESS FOR WORK SAMPLE TEST • Other • Use test results as input to facilitate the interview process and not as a defining step unless you have strong evidence to support a cut score • Use samples of work required on first day of job • Standardize administration and scoring and use of results • Maintain records • • • Test taker name Test taker race and gender Requisition test taker is being considered for Test date Test results – component, overall and any decision resulting from the test © 2014 People. Fluent | Confidential Presentation Title | date 45
Thank You Please contact me with questions or comments lisa. harpe@peoplefluent. com (919) 645 -2975 For more information www. peoplefluent. com Corporate Headquarters (877) 820 -4400 (toll free) or (919) 645 -2800 © 2014 People. Fluent | Confidential Presentation Title | date 46
