PSY 6430 UNIT 6 Validity Determining whether selection

  • Slides: 70
Download presentation
PSY 6430 UNIT 6 Validity Determining whether selection instruments are jobrelated Lecture: ME 1:

PSY 6430 UNIT 6 Validity Determining whether selection instruments are jobrelated Lecture: ME 1: Lecture: Exam: Wednesday, 3/13 Monday, 3/18 Wednesday, 3/20 Monday, 3/25 Last day to withdraw: Monday, 3/18 Task Analysis Workshop: Wednesday, 3/27 1

Task Analysis Workshop 3/27 2 Purpose: To ask me questions about your task statements

Task Analysis Workshop 3/27 2 Purpose: To ask me questions about your task statements and KSAs/WRCs for your project I will not review spelling/grammar or formatting of the task analysis document itself Come prepared with specific questions I will meet with you individually or in small groups Assign times just so you don’t have to wait around too long Email me no later than 3/20 if you plan on (completely optional) (also welcome, of course, to come to my office hours) attending; I will give you a schedule on 3/25

Task Analysis Workshop 3/27 3 You can also, of course, come to my office

Task Analysis Workshop 3/27 3 You can also, of course, come to my office hours and meet with me, if you prefer The task analysis is due on Monday, 4/08 if you want your grade by 4/17 in order to determine if you want to take ME 2 The final due date is Monday, 4/22 at 5: 00 pm Look at the syllabus or U 4 ppt for contingencies regrading lateness

4 SO 1: NFE, Validity, a little review Predictor = test/selection instrument Use the

4 SO 1: NFE, Validity, a little review Predictor = test/selection instrument Use the score from the test to predict who will perform well on the job Possible confusion (again) You need to determine the validity of the test based on your current employees Then you administer it to applicants and select employees based on the score (a few students had a problem distinguishing between validity and reliability on E 5, example next)

SO 1: NFE, Validity, example 5 Administer a test to current employees Obtain measures

SO 1: NFE, Validity, example 5 Administer a test to current employees Obtain measures of how well they perform on the job Correlate the test scores with the performance measures Assume: The correlation is statistically significant Assume: Current employees who score 50 -75 also are performing very well on the job Now you administer the exam to applicants, predicting that those who score 50 -75 will also perform well on the job

SO 2: Reliability vs. Validity 6 Reliability Operational Definition: Is the score on the

SO 2: Reliability vs. Validity 6 Reliability Operational Definition: Is the score on the measure stable, dependable, and/or internally consistent? Conceptual Definition: Are you actually measuring what you want to be measuring? Validity Is the measure related to performance on the job?

SO 3: Relationship between reliability and validity 7 A measure can be reliable, but

SO 3: Relationship between reliability and validity 7 A measure can be reliable, but not valid However, a measure cannot be valid unless it is reliable *Reliability is a necessary but not sufficient condition for validity Text gives a perfect example You can reliably measure eye color, however, it may not be related to job performance at all *key point

Types of validation procedures 8 Content: expert judgment Criterion-related: statistical analyses (concurrent & predictive)

Types of validation procedures 8 Content: expert judgment Criterion-related: statistical analyses (concurrent & predictive) Construct (but not practical-not covering this) Validity generalization (transportable, no local validity study – jobs are similar) Job component validity (not covering this) Small businesses: Synthetic validity (not covering it, not very relevant now –content validity) (main types are the two kinds of criterion-related and content validity; construct really a hold over from test construction - not very relevant - I have only seen this used by a few organizations – create their own tests; cover validity generalization, but right now while validity generalization has excellent professional support, may not be legal - professional guidelines depart from legal; in one case, 6 th Circuit Court ruled it illegal as a matter of law based on Griggs/Duke and Albermarle - 1987)

9 SO 5 NFE but related to 7 A which is: Difference between content

9 SO 5 NFE but related to 7 A which is: Difference between content and criterionrelated validity Content validity relies on expert judgment and a match between the “content” of the job and the “content” of the test Expert judgment refers to the determination of the tasks , KSAs and WRCs required to perform the job via a task analysis and task inventory/questionnaire linking the KSAs and WRCs to selection procedures that measure them (actually covered this in U 4, job analysis, but. . )

10 SO 5 NFE but 7 B is: Difference between content and criterion-related validity

10 SO 5 NFE but 7 B is: Difference between content and criterion-related validity Empirical or criterion-related validity relies on statistical analyses (correlation of test scores with measures of job performance) Measures of job performance = criterion scores Two basic types of criterion-related validity Concurrent validity Predictive validity (content next slide)

Intro to content validity 11 You do NOT use statistical correlation to validate your

Intro to content validity 11 You do NOT use statistical correlation to validate your tests Validation is based “only” on your job analysis procedures and descriptively linking the KSAs and WRCs to selection measures It is much more widely used than criterionrelated validity Particularly since Supreme Court ruled it was OK to use for adverse impact cases (1995) (again, to emphasize)

SO 6: Two reasons why content validity is often used 12 It can be

SO 6: Two reasons why content validity is often used 12 It can be used with small numbers of employees Large sample sizes are required to use criterion-related validity due to the correlation procedures The text later when talking about criterionrelated validity indicates you may need over several hundred Dickinson: usually 50 -100 is adequate How many companies have that many current employees in one position or select that many employees for one position? (small incumbents) number Youofcan do a study over a couple of years but that obviously delays the results of the study

SO 6: Two reasons why content validity is often used 13 Many organizations do

SO 6: Two reasons why content validity is often used 13 Many organizations do not have good job performance measures You need good performance criterion measures to do a criterion-related validity study because you correlate the test scores with job performance measures

SO 7 B: Content vs. criterion-related validity and the type of selection procedure 14

SO 7 B: Content vs. criterion-related validity and the type of selection procedure 14 If you use content validity you should write the test, not select an off-the-shelf test If you use criterion-related validity, you can do either It is much easier and less time consuming to use an off-the-shelf test than to write one! (Covered SO 7 A; VERY IMPORTANT!; book waffles on this a bit, indicating that emphasis should be placed on constructing a test, But only in rare situations would I recommend selecting off-the-shelf test with content validity - legally too risky; why, next slide)

SO 7 B: Why should you write the test if you use content validity?

SO 7 B: Why should you write the test if you use content validity? (this slide, NFE) 15 Content validity relies solely on the job analysis The KSAs must be represented proportionately on the selection test as indicated in the job analysis in terms of: Their relative importance to the job The percentage of time they are used by the employees It is highly unlikely that an off-the-shelf test will proportionately represent the KSAs as determined by your job analysis In some discrimination court cases, the judge has gone through the test item by item to determine whether the items were truly proportional to the KSAs as determined by the job analysis Thus, there is both a professional measurement reason and a legal reason to write the test rather than using an off-the-shelf test

16 SO 9: Major steps of content validity very, very specific requirements for the

16 SO 9: Major steps of content validity very, very specific requirements for the job analysis Determine the tasks for the job *Determine the criticality and/or importance of each of the tasks Specify the KSAs required for EACH task KSAs must be linked to each task (NFE) *Now because of ADA, is it an essential function? (cont. next slide)

SO 9: Major steps of content validity, cont. 17 Determine the criticality and/or importance

SO 9: Major steps of content validity, cont. 17 Determine the criticality and/or importance of each KSA* Describe the relationship between each KSA and each task statement You can have KSAs that are required for only one or two tasks, or you can have KSAs that are required to perform several tasks The more tasks that require the KSAs, the more important/critical they are Describe the complexity or difficulty of obtaining each KSA (formal degree, experience) Specify whether the employee must possess each KSA upon entry or whether it can be acquired on the job (cannot test for a KSA if it can be learned within 6 months) Indicate whether each KSA is necessary/essential for successful performance of the job *Only the first major point will be required for the exam, but I want to stress how detailed your job analysis must be for content validity (cont on next slide)

SO 9: Major steps of content validity, cont. 18 Link important job tasks to

SO 9: Major steps of content validity, cont. 18 Link important job tasks to important KSAs* (FE) Reverse analysis; you have linked the KSAs to the tasks, now you must link the KSAs to the tasks (NFE) KSA # 1 may be relevant to Tasks 1, 6, 7, 10, 12, & 22 KSA # 2 may be relevant to Tasks 2, 4, & 5 Etc. (NFE) Develop test matrix for the KSAs If you want see how you go from the task analysis to the actual test, turn back to pages 91 -96, Figures 3. 17, 3. 18, 3. 19

SO 10: When you can’t use content validity according to the Uniform Guidelines 19

SO 10: When you can’t use content validity according to the Uniform Guidelines 19 When assessing mental processes, psychological constructs, or personality traits that cannot be directly observed, but are only inferred You cannot use content validity to justify a test for judgment, integrity, dependability, extroversion, flexibility, motivation, conscientiousness, adaptability, or any personality characteristic The reason for that is that you are basing your job analysis on expert judgment - and judgment is only going to be reliable if you are dealing with concrete KSAs such as mechanical ability, arithmetic ability or reading blue prints The more abstract the KSA, the less reliable judgment becomes If you can’t see it, if you can’t observe it, then the leap from the task statements to the KSAs can result in a lot of error (review from U 4 text mentions three; I amcan havingrewrite you learn the first trait one and one I addeditin the SOs -- these are the two that Per U 4, you the making are most violated in practice; the second one is relevant to BOTH content and criterion-related so shouldn’t be listed objective/observable under when you can’t use content validity: cannot test for KSAs that can be learned on the job)

SO 10: When you can’t use content validity according to the Uniform Guidelines, cont.

SO 10: When you can’t use content validity according to the Uniform Guidelines, cont. 20 When selection is done by ranking test scores or banding them (from U 1) If you rank order candidates based on their test scores and select on that basis, you cannot use content validity - you must use criterion-related validity If you band scores together, so those who get a score in a specified range of scores are all considered equally qualified, you cannot use content validity - you must use criterion-related validity Why? If you use ranking or banding, you must be able to prove that individuals who score higher on the test will perform better on the job - the only way to do that is through the use of statistics The only appropriate (and legally acceptable) cut-off score procedure to use is a pass/fail system where everyone above the cut-off considered equally (onlyscore relevant ifis adverse impact)

21 SO 11: Conflict between the Uniform Guidelines and selection specialists (293, 2 not

21 SO 11: Conflict between the Uniform Guidelines and selection specialists (293, 2 not 293, 1) As indicated previously the UG state that content validity may not be used to support mental processes-cognitive abilities-such as “intelligence” However, there is general agreement among selection specialists that content validity can be used for general cognitive ability In fact, more recent standards, specifically, the Standards for Educational and Psychological Testing and the Principles for the Validation and Use of Personnel Selection Procedures, state that content validity can be used to support the use of some cognitive ability tests (another one of the discrepancies between the law and the profession; remember the law/judges will follow what is in the UG)

SO 11: Why the conflict? 22 Remember that the Uniform Guidelines were passed/agreed to

SO 11: Why the conflict? 22 Remember that the Uniform Guidelines were passed/agreed to by all of the agencies involved in selection and placement in 1978 Since that time, the courts have used the Uniform Guidelines as a checklist for what is and what is not accepted Once precedents are determined in case law through court rulings (in this case based on the Uniform Guidelines), current and future decisions are based on those precedents/past rulings New knowledge has accumulated since the Uniform Guidelines were approved, however This new knowledge is reflected in the two professional documents, the Standards and Principles, which are (Americanupdated Educational Association and SIOP) by the relevant professional organizations

Criterion-related validity studies: Concurrent vs. predictive 23 SO 12 A: Concurrent validity Administer the

Criterion-related validity studies: Concurrent vs. predictive 23 SO 12 A: Concurrent validity Administer the predictor to current employees and correlate scores with measures of job performance Concurrent in the sense that you have collected both measures at the same time for current employees SO 16 A: Predictive validity Administer the predictor to applicants, hire the applicants, and then correlate scores with measures of job performance collected 6 -12 months later Predictive in the sense that you do not have measures of job performance when you administer the test - you collect them later (comparison of the two, SO 12 A, describe concurrent validity; SO 16 A, describe predictive validity)

24 SO 12 B: Steps for conducting a concurrent validity study Job analysis: Absolutely

24 SO 12 B: Steps for conducting a concurrent validity study Job analysis: Absolutely a legal requirement Discrepancy between law and profession (learn for exam) Law requires a job analysis (if adverse impact & challenged) Profession does not as long as the test scores correlate significantly with measures of job performance Determine KSAs and other relevant requirements from the job analysis, including essential functions for purposes of ADA Select or write test based on KSAs (learn for exam) May select an off-the-shelf test or Write/construct one

25 SO 12 B: Steps for conducting a concurrent validity study Select or develop

25 SO 12 B: Steps for conducting a concurrent validity study Select or develop measures for job performance Sometimes a BIG impediment because organizations often do not have good measures of performance Administer test to current employees and collect job performance measures for them Add this step: Correlate the test scores with the job performance measures Add this step: Determine whether the correlation is statistically significant at the. 05 level You can then use the test to select future job applicant

26 SO 13 A: Advantage of concurrent validity over predictive validity Because you are

26 SO 13 A: Advantage of concurrent validity over predictive validity Because you are using the test data and performance data from current employees, you can conduct the statistical validation study quickly – in a relatively short period of time Remember, that with predictive validity, you must hire applicants and then wait 6 -12 months to obtain measures of job performance (post-training, after they have learned the job)

27 SO 13 B: The basic reason that accounts for all of the weaknesses

27 SO 13 B: The basic reason that accounts for all of the weaknesses with concurrent validity All of the weaknesses have to do with differences between your current employees and applicants who apply for the job in the future You are conducting your study with one sample of the population (your employees) and assuming conceptually that your applicants are from the same population However, your applicants may not be from the same population - they may differ in important ways from your current employees Ways that would cause them (as a group) to score differently on the test or perform differently on the job, affecting the correlation (job relatedness) of the test (text lists several weaknesses and all of them really relate to one issue; dealing with inferential statistics here; as per the SO, only partial credit for the first bullet))

SO 14: Restriction in range 28 This is the term used for the statistical/mathematical

SO 14: Restriction in range 28 This is the term used for the statistical/mathematical reason why the differences between your current employees and applicants affect validity It also explains from the last unit, why reliability is generally higher when Your sample consists of individuals who have greater differences in the ability for which you are testing: High school students, community college students, vs. engineering majors in college who take a math test The questions are moderately difficult – about 50% of test takers answer the questions correctly – rather then when the questions are very easy or very difficult (so I have talked about this before, I just haven’t named it)

SO 14: Restriction in range 29 With criterion-related validity studies the ultimate proof that

SO 14: Restriction in range 29 With criterion-related validity studies the ultimate proof that your selection test is job related is that the correlation between the test scores and job performance measures is statistically significant A high positive correlation tells you People who score well on the test also perform well People who score middling on the test are also middling performers People who score poorly on the test also perform poorly on the job In order to obtain a strong correlation you need People who score high, medium, and low on the test People who score high, medium, and low on the performance measure

SO 14: Restriction in range, cont. 30 That is, you need a range of

SO 14: Restriction in range, cont. 30 That is, you need a range of scores on BOTH the test and the criterion measure in order to get a strong correlation If you only have individuals who score about the same on the exam, and some perform well, middling, and poorly, you will get a zero correlation (makes sense, you can’t predict performance from the test) Similarly if you have individuals who score high, medium, and low on the test, but they all perform about the same, you will get a zero correlation (again, it makes sense, you can’t predict performance from the test)

SO 14: Restriction in range, cont. 31 Any procedure/factor that decreases the range of

SO 14: Restriction in range, cont. 31 Any procedure/factor that decreases the range of scores on either the test or the performance measure Reduces the correlation between the two and, hence, Underestimates the true relationship between the test and job performance That is, you may conclude that your test is NOT valid, when in fact, it may be

SO 14: Restriction in range, cont. 32 Restriction in range is the technical term

SO 14: Restriction in range, cont. 32 Restriction in range is the technical term for the decrease in the range of scores on either or both the test and criterion Concurrent validity tends to restrict the range of scores on BOTH the test and criterion, decreasing the correlation between the two, and hence underestimating the true validity of a test In other words, to be repetitive, this increases the possibility that you will conclude that your test is not valid, when in fact, it is (finally - the definition; stress the either or both; cont on next slide)

33 SO 14: Restriction in range, cont. Also related to SO 15 Why? You

33 SO 14: Restriction in range, cont. Also related to SO 15 Why? You are using current employees in your sample Your current employees have not been fired because of poor performance (only ”good” employees are left) Your current employees have been doing the job for a while and thus are more experienced All of the above would be expected to Result in higher test scores than for the population of applicants Result in higher performance scores than for the population Thus, restricting the range of scores on both the test and the performance criterion measure (diagrams on next slide)

SO 14: Restriction in range, cont. 34 Top diagram restriction in range Strong correlation

SO 14: Restriction in range, cont. 34 Top diagram restriction in range Strong correlation Performance No High Low Bottom diagram Restriction in range Test scores and Performance scores Zero correlation Low High Test Scores High Performance Low High Test Scores (extreme example, but demonstrates point - concurrent validity is likely to restrict range on both, underestimating true validity)

SO 16: Predictive validity 35 SO 18 A: Predictive validity (review) Administer the predictor

SO 16: Predictive validity 35 SO 18 A: Predictive validity (review) Administer the predictor to applicants, hire the applicants, and then correlate scores with measures of job performance collected 6 -12 months later Predictive in the sense that you do not have measures of job performance when you administer the test - you collect them later, hence, you can determine how well your test actually predicts future performance

SO 16 B: Steps for a predictive validity study 36 Job analysis: Absolutely a

SO 16 B: Steps for a predictive validity study 36 Job analysis: Absolutely a legal requirement Determine KSAs and other WRCs from the job analysis, including the essential functions for purposes of ADA Select or write test based on KSAs* You may select an off-the-shelf test or Write/construct one Select or develop measures for job performance *Learn this point for the exam (first four steps are exactly the same as for a concurrent validity study)

SO 16 B: Steps for a predictive validity study 37 Administer the test to

SO 16 B: Steps for a predictive validity study 37 Administer the test to job applicants and select randomly or using the existing selection system Do NOT use the test scores to hire applicants (I’ll come back to this later) After a suitable time period, 6 -12 months, collect job performance measures Add this step: Correlate the test scores with the performance measures Add this step: Determine whether the correlation is statistically significant and if it is, your test is valid

SO 17: Two weaknesses of predictive validity (299, 2 not 299, 3) 38 Time

SO 17: Two weaknesses of predictive validity (299, 2 not 299, 3) 38 Time it takes to validate the test Need appropriate time interval after applicants are hired before collecting job performance measures If the organization only hires a few applicants per month, it may take months, a year, or even more than a year to obtain a large enough sample to conduct a predictive validity study (N=50 -100)

SO 17: Two weaknesses of predictive validity 39 Very, very difficult to get managers

SO 17: Two weaknesses of predictive validity 39 Very, very difficult to get managers to ignore the test data (politically very difficult) Next to impossible to get an organization to randomly hire - some poor employees ARE going to be hired Also difficult to convince them to hire using the existing selection system without using the test score (but much easier than getting them to randomly hire and doable) I love the reported conversation between one of the authors and a VP of personnel included in the text. (I don’t blame them; it would be like us randomly accepting students into the graduate program)

Predictive Validity: Three basic ways to do it 40 Pure predictive validity: by far

Predictive Validity: Three basic ways to do it 40 Pure predictive validity: by far the best Administer the test to applicants and randomly hire Current system: next best, more practical Administer the test to applicants, use the current selection system to hire (NOT the test) Use test to hire: bad, bad both professionally and legally Administer the test, and use the test scores to hire applicants (going to come back to these and explain the evaluations; text lists the third as an approach! Click: NO!!)

SO 18: Predictive validity designs 41 Figure 8. 5 lists 5 types of predictive

SO 18: Predictive validity designs 41 Figure 8. 5 lists 5 types of predictive validity designs Follow-up: Random selection (pure predictive validity) Best design No problems whatsoever from a measurement perspective; completely uncontaminated from a professional perspective Follow-up: Use present system to select OK and more practical, but It will underestimate validity if your current selection system is valid; and the more valid it is the more it will (answer not on slide) underestimate the validity of your test

SO 18: Predictive validity, selection by scores 42 Select by test score: Do NOT

SO 18: Predictive validity, selection by scores 42 Select by test score: Do NOT do this!!! Professional reason: If your selection procedure is job related, it will greatly underestimate your validity - and, the more job related the selection procedure is, the greater it will underestimate validity. Why? If your test is valid, you are severely restricting the range on both your test and your job performance measures! Thus your correlation will be much smaller than it should be And, you are likely to conclude that your test is not valid when in fact it is (professional and legal reasons not to do this)

SO 18: Predictive validity, selection by scores 43 Legal reason: If adverse impact occurs

SO 18: Predictive validity, selection by scores 43 Legal reason: If adverse impact occurs you open yourself up to an unfair discrimination law suit You have adverse impact, but you do not know whether the test is job related There is a caveat (NFE): Some courts have ruled that adverse impact is OK if a validation study is in progress. However, I see this as being way too risky legally (particularly given the technical problems with this method).

44 SO 18: NFE, Further explanation of types of predictive validity studies from Figure

44 SO 18: NFE, Further explanation of types of predictive validity studies from Figure 8. 5 Hire, then test and later correlate test scores and job performance measures If you randomly hire, this is no different than pure predictive validity: #1 previously, Follow-up: Random selection If you hire based on current selection system, this is no different than #2 previously, Follow-up: Select based on current system (both are perfectly acceptable; one more slide on this)

SO 18: NFE, Further explanation of types of predictive validity studies 45 Personnel file

SO 18: NFE, Further explanation of types of predictive validity studies 45 Personnel file research - applicants are hired and their personnel records contain test scores or other information that could be used as a predictor (i. e. , perhaps from a formal training program). At a later date, job performance measures are obtained.

For exam: Rank order of criterion-related validity studies in terms of professional measurement standards

For exam: Rank order of criterion-related validity studies in terms of professional measurement standards 46 1. 2. 5 Predictive validity (pure) - randomly hire Predictive validity – use current selection system 2. 5 Concurrent validity 4. Predictive validity – use test scores to hire

SO 19: Which is better: Predictive vs. concurrent, research results 47 Data that exist

SO 19: Which is better: Predictive vs. concurrent, research results 47 Data that exist suggest that: Concurrent validity is just as good as predictive validity for ability tests (most data) May not be true for other types of tests such as personality and integrity tests Studies have shown differences between the two for these type of tests – as well as for structured interviews and biographical data inventories - so proceed with caution! (Conceptually, predictive validity is better, it has more fidelity with, is more similar to the actual selection procedure; test applicants, select, and see how well they do on the job later)

SO 20: Sample size needed for a criterion-related validity study (review) 48 Large samples

SO 20: Sample size needed for a criterion-related validity study (review) 48 Large samples are necessary The text doesn’t give a “magic” number, understandably A sample of 100 is usually adequate – and even 50 is possible (but risky): learn Dickinson’s number What do companies do if they do not have that many employees? They use content validity They could possibly also use validity generalization, but even though this would be professionally acceptable, at the current time it is still legally risky

Validation procedures 49 Content: expert judgment (no statistical analysis) Criterion-related: statistical analyses (concurrent &

Validation procedures 49 Content: expert judgment (no statistical analysis) Criterion-related: statistical analyses (concurrent & predictive) Concurrent: Correlate test scores with performance measures for current employees Predictive: Correlate test scores for individuals you hire with performance measures for those individuals 6 -12 months later Construct (but not practical-not covering this on the exam) Validity generalization (transportable, no local validity study – jobs are similar)

SO 21: NFE, Construct validity 50 Every selection textbook covers construct validity I am

SO 21: NFE, Construct validity 50 Every selection textbook covers construct validity I am not covering it for reasons indicated in the study objectives Basic reason for not covering it is that while construct validity is highly relevant for test construction, very few organizations use this approach - it’s too time consuming and expensive First, the organization develops a test and determines whether it is really measuring what it is supposed to be measuring Then, they determine whether the test is job

SO 26: Validity generalization, what it is 51 Validity generalization is considered to be

SO 26: Validity generalization, what it is 51 Validity generalization is considered to be a form of criterion-related validity, but you don’t have to conduct a “local” validity study, that is, you don’t have to conduct a validity study in your organization using your employees Rather you take validity data from other organizations for the same or very similar positions and use those data to justify the use of the selection test(s) Common jobs: computer programmers and systems analysts, set-up mechanics, clerk typists, sales representative, etc. (I am skipping to SO 26 for the moment, SOs 24 -26 relate to statistical concepts about correlation; organization of this chapter Is just awkward. I want to present all of the validity procedures together, and then compare them with respect to when you should/can use one or the other. Then, I’ll return to SOs 22 -25: cont on next slide)

SO 26: Validity generalization, what it is 52 1. 2. At least two types

SO 26: Validity generalization, what it is 52 1. 2. At least two types of studies/information are required Obtain studies that indicate that a selection measure is valid for similar jobs in other settings Provide data showing that the job in your organization is similar to the jobs in the other organizations An analysis of the job in the current setting, your organization, is mandatory and you must compare it to the job analysis in the other organizations

SO 26: Validity generalization, what it is 53 Assumption is that those data will

SO 26: Validity generalization, what it is 53 Assumption is that those data will generalize to your position and organization Thus, you can use this approach if you have a very small number of employees and/or applicants* *Note this point well

SO 27: Validity generalization, cont. 54 Testing experts completely accept the legitimacy of validity

SO 27: Validity generalization, cont. 54 Testing experts completely accept the legitimacy of validity generalization Primarily based on the stellar work of Schmidt and Hunter (who was a professor at MSU until he retired) Gatewood, Feild, & Barrick believe this has a bright future Frank Landy (also a legend in traditional I/O) is more pessimistic about it Wording of the CRA of 1991 may have made this illegal There has not been a test case No one wants to be the test case (you should not be the test case) (this slide, NFE, cont. on nxt slide)

SO 27: Validity generalization, cont. 55 Actually have come full circle with respect to

SO 27: Validity generalization, cont. 55 Actually have come full circle with respect to validity generalization and its acceptance by testing specialists In the early days of testing, validity generalization was accepted If a test was valid for a particular job in one organization it would be valid for the same or a similar position in another organization It then fell into disfavor, with testing specialists reversing their position, and adhering to situational specificity Now, based on Schmidt and Hunter’s work, it is again embraced by testing specialists

SO 28 Why validity generalization is not accepted by the courts 56 The Uniform

SO 28 Why validity generalization is not accepted by the courts 56 The Uniform Guidelines, precedent-setting court cases, and the CRA of 1991 state that a validity study must be conducted with the organization’s own applicants or employees and measures of their actual job performance Validity generalization is conducted with the applicants or employees of other organizations Using validity studies/data that include the job performance of those applicants and employees, not the organization’s own applicants and employees

SO 31: Interesting fact (and for the exam) 57 In a 1993 random survey

SO 31: Interesting fact (and for the exam) 57 In a 1993 random survey of 1, 000 organizations listed in Dun’s Business Rankings with 200 or more employees, the percentage of firms indicating that they had conducted validation studies of their selection measures was: 24% In today’s legal environment, the other organizations could find themselves in a whole world of hurt! (granted, old data and I couldn’t find any new data; click, click!)

58 Factors that affect the type of validity study: When to use which validity

58 Factors that affect the type of validity study: When to use which validity strategy Four main factors that influence the type of validity study you can do Sample size Cut-off score procedures Type of attribute measured: observable or not Type of test: write or off-the-shelf (on the exam, I am likely to give you situations and ask you, given the situation, what type of validity strategycould you use and why: That is, what options do you have? That’s exactly the type of decision you are going to have to make in organizations. So, to make it easier, and summarize things: Include validity generalization in your answers; BE SURE TO STUDY THIS MATERIAL

59 Factors that affect the type of validity study: When to use which validity

59 Factors that affect the type of validity study: When to use which validity strategy Sample size Large # employees Concurrent (all forms, OK) Predictive Content Validity generalization Small # employees Content Validity generalization (it’s OK to use content and validity gen with large sample sizes; many orgs do use content!)

60 Factors that affect the type of validity study: When to use which validity

60 Factors that affect the type of validity study: When to use which validity strategy Cut-off score procedures Minimum (pass/fail) Concurrent (all forms, OK) Predictive Content Validity generalization Ranking or banding (only criterion-relatedall but content) Concurrent Predictive Validity generalization (validity generalization is based on correlation, even if you don’t do the study yourself, so remember it is considered a type Of criterion-related study)

61 Factors that affect the type of validity study: When to use which validity

61 Factors that affect the type of validity study: When to use which validity strategy Attribute being measured Observable Concurrent (all forms, OK) Predictive Content Validity generalization Not observable (only criterion-relatedall but content) Concurrent Predictive Validity generalization (personality, extraversion, social sensitivity, flexibility, integrity, etc. ; cogntivie ability – intelligence – gray area)

62 Factors that affect the type of validity study: When to use which validity

62 Factors that affect the type of validity study: When to use which validity strategy Type of test Write/construct Concurrent (all forms, OK) Predictive Content Validity generalization Off-the-shelf (only criterion-relatedall but content) Concurrent Predictive Validity generalization (next slide, back to SO 23; interpretation of validity correlation)

SO 23: Statistical interpretation of a validity coefficient 63 Recall, rxy = correlation coefficient

SO 23: Statistical interpretation of a validity coefficient 63 Recall, rxy = correlation coefficient r 2 = coefficient of determination Coefficient of determination: The percentage of variance on the criterion that can be explained by the variance associated with the test r =. 50, to statistically interpret it: r 2 =. 25 25% of the variance on job performance can be explained by the variance on the test Less technical, but OK 25% of the differences between individuals on the job performance measure can be accounted for by differences in their test scores

SO 23: Validity vs. reliability correlations 64 You interpret a validity correlation coefficient very

SO 23: Validity vs. reliability correlations 64 You interpret a validity correlation coefficient very differently than a reliability correlation coefficient You square a validity correlation coefficient You do NOT square a reliability correlation coefficient Why? A bit of review With a reliability correlation coefficient you are basically correlating a measure with itself Test-retest reliability Parallel or alternate form reliability Internal consistency reliability (split half) (I am not going to go into the math on that to prove that to you)

SO 24: Validity vs. reliability correlations, examples for test 65 You correlate the test

SO 24: Validity vs. reliability correlations, examples for test 65 You correlate the test scores from a mechanical ability test with a measure of job performance The resulting correlation coefficient is. 40 How would you statistically interpret that? 16% of the differences in the job performance of individuals can be accounted for by the differences in their test scores (note carefully, you do not multiply it by two, you square it!)

SO 24: Validity vs. reliability correlations, examples for test 66 You administer a computer

SO 24: Validity vs. reliability correlations, examples for test 66 You administer a computer programming test to a group of individuals, wait 3 months and administer the same test to the same group of individuals. The resulting correlation coefficient is. 90 How do you statistically interpret that correlation coefficient? 90% of the differences in the test scores between individuals are due to true differences in computer programming and 10% of the differences are due to error

67 NFE: Different types of correlation coefficients: or why it is a good idea

67 NFE: Different types of correlation coefficients: or why it is a good idea to take Huitema’s correlation and regression The most common type of correlation to use is the Pearson product moment correlation However, you can only use this type of correlation if You have two continuous variables, e. g. , a range of scores on both x and y If the relationship between the two variables is linear Some have shown a curvilinear relationship between intelligence test scores and performance of sales representatives

68 Different types of correlation coefficients: or why it is a good idea to

68 Different types of correlation coefficients: or why it is a good idea to take Huitema’s correlation and regression Point biserial coefficient is used when one variable is continuous and the other is dichotomous High school diploma vs. no high school diploma (X) Number of minutes it takes a set-up mechanic to set up a manufacturing line (Y) x is dichotomous, y is continuous Phi coefficient is used when both variables are dichotomous High school diploma or no high school diploma (X) Pass or fail performance measure (Y) Both x and y are dichotomous (one more slide on this)

69 Different types of correlation coefficients: or why it is a good idea to

69 Different types of correlation coefficients: or why it is a good idea to take Huitema’s correlation and regression Rho coefficient - Spearman’s rank order correlation when you rank order both x and y, and then correlate the ranks Rank order in test scores Rank order number of minutes it takes set-up mechanics to set up a manufacturing line Use rank order when either your x or y scores are not normally distributed - that is, when there a few outliers - either very high scores on either or very low scores on either (last slide)

END OF UNIT 5 Questions? Comments? 70

END OF UNIT 5 Questions? Comments? 70