Schedule Monday and Wednesday Lecture Monday 408 Exam

Schedule Monday and Wednesday: Lecture Monday, 4/08: Exam Monday, 4/08: Task analysis due if you want your grade by 4/17 to determine whether to take ME 2 1

Mental Measurements Yearbook (1938) Now in its 20 th edition, updated in 2016 You can access it online for free through our library Szostek & Hobson (2011) I included this article so you can get an idea of how important this resource is from a legal perspective. Courts have acknowledged it as the “bible of testing” and the “authoritative source on testing” I also included it because it uses the Myers-Briggs Inventory as an example of an extremely popular, yet really bad selection instrument (Just a brief mention of the MMY – you should always check the reviews for any off-the-shelf test an organization is planning on using – not going over the other study objectives on the article; and yes, it’s a sad thing the author spelled It wrong and the editors did not catch it. I am spelling it correctly) 2

MBTI Personality inventory that classifies individuals into one of 16 types of personalities, using major categories of extraversion/ introversion, intuition/sensing, feeling/thinking, perception/judgment 89 of Fortune 100 companies have used it for selection and/or promotion Over 2 million were administered last year ▪ Just about every executive in business knows his/her MB scores/indicators It has no validity or reliability!! ▪ In terms of reliability, if you retake the inventory after 5 weeks, there’s a 50% chance that you will fall into a different personality category (and, remember, validity is bounded by reliability) And, here is the clincher – it is based on Jungian psychology. No psychologist worth his/her salt takes Jungian psychology seriously any more!! (how many of you have heard of this? Magical mystery tour; Di. SC popular substitute) 3

On the service, it appears that the instrument is useful in personal or professional development. However, the test suffers from questionable reliability and unknown validity. The measure is remiss in its lack of reliability and validity for diverse ethnic/racial populations and professional occupations. The substantiating evidence does not meet the criteria established in the Standards for Educational and Psychological Testing. Therefore, the use of the Di. SC is not recommended. 4

These are perfect examples of companies using selection procedures that are bunk, yet are very, very popular. There are many, many more examples Just because something is popular and used by Fortune 100 companies doesn’t mean it has any professional merit whatsoever. Interestingly, the owners and creators of the MBTI state that “It is unethical to use the MBTI tool for hiring…. The MBTI tool can’t tell you who to hire, but it can help you work with your team so that everyone gives his or her best performance. ” https: //www. cpp. com/pdfs/MBTI_myths_infographic. pdf 5

For decades, tests have been classified as either achievement tests or aptitude tests Definitions of “achievement” and “aptitude” Achievement The act of accomplishing or finishing something successfully, especially by means of skill, practice, or perseverance Aptitude A natural genetic or acquired talent or ability or inclination; quickness in learning and understanding - intelligence The distinction represents the mind-body dualism typical of traditional testing (in this material, GFB argue that the terms “achievement test” and “aptitude test” are inappropriate and should be replaced with the term “ability test” - and I agree with them; excellent material) 6

Achievement tests (supposedly) measure What a person learned as a result of a specific structured educational/training experience/course Scores are interpreted to be a measure of how much an individual knows as a result of the education or training ▪ English grammar, math, science, social studies, etc. These are the types of tests used in grade school and high school to measure student learning/proficiency ▪ In Michigan, MEAP tests: Michigan Educational Assessment Program 7

Aptitude tests (supposedly) measure Natural ability combined with an accumulation of learning from a number of diverse and informal learning experiences Although not emphasized by GFB, there is a genetic implication ▪ ▪ You have artistic ability or you don’t You have mechanical aptitude or you don’t Women don’t have an aptitude for math Men don’t have good spatial aptitude, thus they can’t find their way around the mall Said to measure potential to learn, or the potential to develop new skills and acquire new knowledge ▪ If you don’t have the aptitude you can’t be a good artist, mechanic, mathematician Intelligence tests, SATs, GREs, Artistic Aptitude These are the tests that you are told you can’t study for (nonsense- most people don’t say that any more) (Olympic athletes and musicians have “natural” ability – then we learn the parent was an Olympic athlete or musician – both parents were musicians…Harvard president 2006; math and sciences) 8

All tests measure what a person has learned up to the time he or she takes the test and that is the only thing a test can measure They cannot and do not measure innate or unlearned potential (even if that existed) Thus, the distinction between achievement tests and aptitude tests is arbitrary and We should use the term “ability tests” for both types of tests Ability in the sense of competence or proficiency, regardless of how you have acquired the ability/skill 9

Tests can and do measure the prerequisites that are necessary for further learning in an specified area, and thus can predict future learning/performance If students do not do well in PSY 3600, Concepts and Principles of Behavior Analysis, they cannot do well in PSY 4600, Survey of Behavior Analysis Research, thus a student’s grade in PSY 3600 can predict his or her performance in PSY 4600 You can’t balance an equation in chemistry unless you know algebra, thus a test of algebra can predict performance in a chemistry class (not in text, but important to understand) 10

Mental/cognitive ability tests were at the center of early critical Supreme Court decisions regarding unfair discrimination Thus, many companies stopped using them However, there is a lot of indisputable research in selection that indicates that mental ability tests are related to almost all jobs Validity correlations are often quite high, and higher than other tests Many companies are now using them again Remember, however, if you use one of these, you run a legal risk if you support them via content validity although personnel selection specialists say that is OK 11

10 A: Cognitive ability tests measure the ability to learn in formal education and training situations and scores on these tests correlate highly with scholastic measures of success (506, 0 not 506, 1) 12

A rose is not a rose A cognitive ability test is not a cognitive ability test Cognitive ability tests measure a collection of abilities learned repertoires – that typically include verbal, math, memory, and reasoning abilities 14 different abilities are often measured in some combination by these tests (next slide) Because different tests often measure a different set of these abilities, a person may score differently on different cognitive ability tests 13

Memory span Numerical fluency Verbal comprehension Conceptual classification Semantic relations General reasoning Conceptual foresight Figure classification Spatial orientation Visualization Intuitive Reasoning Ordering Figure identification Logical evaluation and deduction 14

The term cognitive ability makes it explicit that these tests measure various abilities of the applicant, not some innate, unlearned, hypothetical construct called “intelligence” They should be thought of the same way the other abilities discussed in the book are thought of e. g. , mechanical ability, clerical ability In other words, the authors are resisting the traditional view that there is something called “intelligence” 15

The authors describe the Wonderlic Personnel Test which is probably the most popular Given to all players at the NFL Scouting Combine and scores are reported to NFL teams before the annual draft If you go to footballiqscore. com and take one of the tests, you can compare your score to those of “famous football players”! For a moment, look at items in the text that are similar to the ones on the Wonderlic Personnel Test 16

1. Which of the following months has 30 days? (a) February (b) June (c) August (d) December 2. Alone is the opposite of: (a) happy (b) together (c) single (d) joyful 3. Which is the next number in this series: 1, 4, 16, 64, 256, (a) 4 (b) 16 (c) 64 (d) 1024 (Two slides - Note: all six items are different types of items: general knowledge, opposites - verbal comprehension and vocabulary, numerical reasoning and ordering) 17

4. Twilight is to dawn as autumn is to: (a) winter (b) spring (c) hot (d) cold 5. If Bob can outrun Junior by 2 feet in every 5 yards of a race, how much ahead will Bob be at 45 yards? (a) 5 yards (b) 6 yards (c) 10 feet (d) 90 feet 6. The two words relevant and immaterial mean: (a) the same (b) the opposite (c) neither same nor opposite (again, notice the type of questions: semantic or verbal reasoning, numerical fluency/reasoning, verbal comprehension - opposites) 18

What have the validity studies uniformly concluded? Cognitive ability tests are among the most valid of all selection instruments (work samples are the only tests that seem to be as valid, recent data suggest they have just as much adverse impact; next slide on validity of mental ability tests as well) 19

The validity correlations for both cognitive ability tests and other types of tests are highly stable across organizations (and, next slide) 20

Differences in the actual tasks that a person performs as part of a job have very little effect on the magnitude of the validity coefficients for cognitive ability tests In other words, cognitive ability tests are valid predictors of performance for a wide variety of jobs 21

Large differences exist in mean scores between white and black respondents and between white and Hispanic respondents In other words, they do have adverse impact That means that using these tests in selection decisions usually results in a higher percentage of whites being selected than blacks and Hispanics But remember, this does not mean the tests unfairly discriminate against blacks and Hispanics – if the tests are job-related, it is “fair” discrimination NFE, but Asians typically score higher than whites; hence the current controversy at Harvard (and perhaps Yale) re admissions: consideration of race in admissions favors blacks and whites to the disadvantage of Asians 22

17 A: What is meant by differential validity? Notion/hypothesis that tests are less valid for minority groups than for non-minorities ▪ That is, a test may be significantly more valid for whites than for blacks ▪ Term is related to test bias regarding ability tests, particularly cognitive ability tests ▪ This claim is made over and over again with respect to SATs and GREs - that those tests are more predictive of the performance of white students than they are of the performance of minority students (extremely important; and mentioned often in selection as well as admissions to colleges and universities, - and is still very controversial; the notion of differential validity was an attempt to explain the differences on cognitive ability tests) 23

The argument is that the content of ability tests is based on content/items related to the white middle-class (e. g. , vocabulary and grammar), and thus the scores of the minorities are lower than what they should be 24

The data are very clear about this issue Differential validity does not exist • That is, tests are equally valid for whites and other ethnic/racial groups • It makes sense – Verbal comprehension skills are verbal comprehension skills – Verbal reasoning skills are verbal reasoning skills – Math skills are math skills, etc. • Thus if any of these skills are required by the job, they should be “equally required” by whites and members of other ethnic/racial groups 25

What have organizations done to increase diversity given that they want to use cognitive ability tests because of their validity and low cost? Emphasized the recruitment of minorities so there are larger applicant pools Increased the training of minorities who are employed so that there is rapid development of skills that are important for job success But note, that neither of these are ”professional selection issues” that a typically part of a selection specialist’s responsibility (now back to SO 19: remember race norming is not legal; often a problem because selection specialists are typically not the ones who are responsible for recruitment –selection specialists really need to work with the HR staff) 26

The authors describe several very popular tests Refer to this material if you are ever looking for tests in these categories I am not going to have you learn anything specific about these tests 27

The data and information on personality tests is difficult The authors feel that there is reason to be optimistic about their use, but are cautious about their use at the present time Regardless their use is rapidly increasing Just over 5 years ago 30%-40% of companies reported using them to screen job applicants; about 60%-70% of companies report using them today – mainly large organizations (such as Fortune 100 companies) I remain cautious about their use and would like to see more data, but admit I am more positive about their use than I was 5 years ago…. we shall see One of the problems I have, particularly as a behavior analyst, is defining “personality” and thus, how to measure to it but progress is being made…. 28

There is some agreement in the field that personality characteristics can be grouped into five broad dimensions called the Five-Factor Model or Big Five Conscientiousness ▪ Being responsible, organized, dependable, planful, willing to achieve, and persevering Emotional stability (only one described in negative terms) ▪ Being emotional, tense, insecure, nervous, excitable, apprehensive, and easily upset Agreeableness (relevant for team work) ▪ Being courteous, flexible, trusting, good natured, cooperative, forgiving, softhearted, and tolerant Extroversion ▪ Being sociable, gregarious, assertive, talkative, and active Openness to experience (also called intellect or culture) ▪ Being imaginative, cultured, curious, intelligent, artistically sensitive, original and broad minded 29

More on this later, but recent meta-analyses do show that these traits can be relevant predictors of broad criteria of work effectiveness and retention of productive workers And, while not as important, this is something that managers believe is very important, nearly as important as general cognitive ability – thus, use of personality tests as part of the selection process can increase support of a selection system by hiring managers And, this is never, ever a bad thing! 30

There are two additional important reasons that argue in favor of using personality tests Personality is not highly correlated with other useful selection tools (cognitive ability tests, etc. ) and thus contribute incremental validity to the prediction of successful performance – above and beyond what other selection tools do There is little or no adverse impact ▪ Mean scores are comparable across racial and ethnic groups ▪ There is some difference between males and female at the individual trait level (just looking at one or two of the traits) but a smaller difference at the overall Five Factor Level (that is, when all of the factors are taken into account) 31

23 A: In a very large meta-analysis study conducted by one of the authors (Barrick), conscientiousness and emotional stability were valid predictors of overall work performance over all of the students examined This led them to conclude that these traits were universal predictors across all jobs (very strong conclusions!) 23 B: The composite Five Factors predict (a) career success (status and income) 50 years later (when personality is assessed in high school) and (b) retention of productive employees by three of the traits (conscientiousness, emotional stability, and agreeableness) (I am going to skip to SO 23 and then return to SO 22, that deals with emotional intelligence; obviously, great promise) 32

If you do use a personality test, you must use a criterionrelated validity study to support it because personality traits cannot be directly observed Concurrent validity Predictive validity Validity generalization (in other words you cannot use content validity: also have some legal issues to be aware of) 33

Emotional Intelligence is one of the latest hypes It has been lauded by the Harvard Business Review “Today companies worldwide routinely look through the lens of EI in hiring, promoting, and developing their employees. ” Toy companies now advertise toys and games that will increase a child’s EI There are no data to support it (once again)!! The most interesting aspect to EI is that we don’t know what it is Research in work settings has been insufficient to conclude that it is merit 34

The most interesting aspect to EI is that we don’t know what it is, and various EI surveys include various aspects of what the authors of the surveys include as ”EI” Research in work settings has been insufficient to conclude that it has merit “The ratio of hyperbole to hard evidence is rather high, with overreliance on anecdotes and unpublished surveys” The authors state: “It is incumbent upon the practitioner to carefully examine evidence of predictive validity and other psychometric evidence to ensure that the EI measure relates to important outcomes at work. We believe the field is not currently at an adequate level of understanding for EI to be recommended as a selection tool…. ” 35

If a test can and is used to diagnose mental and psychiatric disorders, then it will probably be considered a medical examination under ADA and can only be administered postoffer (we dealt with this a bit in U 3 as well) If it deals with other personality traits (the Big 5, for example) then it probably will not be considered a medical examination although I don’t know how courts would/will handle “emotional stability” as it relates to ADA Nonetheless, my strong advice to you is to treat every personality test as a medical examination until things are clarified more by the courts Which means you should only administer personality tests postoffer and keep the results in a file that is separate from the personnel file (back to personality tests; include material in italics – this is important) 36

Clarifying court case, 2005, Seventh Circuit Court MMPI is a medical examination and thus illegal for preoffer use in selection (certainly that was expected) Psychological tests that measure personal traits such as honesty, integrity, preferences and habits do not constitute medical examinations 37

Right to privacy: Although a right to privacy is not explicitly guaranteed under the U. S. Constitution, individuals are protected from unreasonable intrusions and surveillance Personality tests, by their nature, reveal an individual’s thoughts and feelings Several states have laws that explicitly guarantee a right to privacy ▪ To date, litigation has occurred about questions relating to religious views and sexual inclinations, orientation, and identification 38

Soroka v. Dayton Hudson (1991) California Court of Appeals stopped Dayton Hudson’s Target stores from requiring applicants for store security positions to take a personality test that contained questions about sexual practices and religious beliefs The court also stated that employers must restrict psychological testing to job-related questions The ruling was later dismissed because the parties reached a courtapproved settlement ▪ Dayton-Hudson agreed to stop using the personality test ▪ Divided $1. 3 million dollars among the estimated 2, 500 members of the plaintiff class who had taken the test (last slide on this) 39

Simulation or work sample tests are excellent and I highly recommend their use when you can do them Typing test Having candidates write a computer program to solve a specific problem Role playing a sales situation with an applicant for a sales position Having mechanics trouble shoot a problem with an engine You are getting an actual sample of behavior under controlled testing conditions (which permits you to easily compare performance across applicants) 40

Difficulty of accurately simulating job tasks that are representative of the job This is particularly true for managerial and professional jobs Applicants must already have the KSAs being tested – they cannot cover specialized things that must be learned on the job General sales skills OK, but questions that deal with specific company- related products and pricing are not You can allow an applicant to “study” that type of material before the exam, if that type of skill is a requirement of the job, but care must be taken Very costly to develop and often to administer (many must be done one-on-one) (stress interview good example of a nonexample; commonly used, and have been used by behavioral consulting firms; high stress – interview, rapidly ask questions, interrupt applicant, be semi-hostile; rare that type of stress is what is encountered in a work setting – unless you are before a congressional committee or are a press secretary for a president!) 41

Validity They both have high validity: they are two of the most valid types of selection instruments Adverse impact Equal adverse impact Cost Performance tests cost much more to develop and administer 42

Assessment centers or even the use of some of the exercises often included in assessment centers have been highly successful In-basket tests Leaderless group interaction tests Case analyses Main problem is their time and expense to both develop and administer You are unlikely to become involved in designing an assessment center, thus I am skipping them for the sake of time 43

Refer you to the Minnich & Komaki article in U 7 in the course pack from the OBM Network News The article describes the use of a validated in-basket test to assess the effectiveness of managers based on Komaki’s Operant Supervisory Taxonomy and Index This is one of the best examples I have ever seen of the intersection of behavior analysis and traditional I/O Psychology Operant supervisory taxonomy and index Assessed the difference between high performing and low performing managers Found that work sampling and type of consequence following performance distinguished between high and low performing managers (Gives a detailed description of the instrument, some of the actual items, and responses, along with analysis of responses Unfortunately, it is not commercially available – done as Minnich’s dissertation) 44

For all practical purposes, it is illegal Federal law, Employee Polygraph Protection Act of 1988 It can be used in some specific employment situations for selection (there are other requirements for use with current employees) ▪ Private employers whose primary business purpose is to provide security services (e. g. , protection of nuclear power facilities, public water supply facilities, shipments or storage of radioactive or other toxic waste materials, public transportation) ▪ Employers involved in the manufacture, distribution, or dispensing of controlled substances ▪ Federal, state and local government employers; also private consultants or experts under contract to governmental depts. and agencies (e. g. , Defense Dept. , Energy Dept. , National Security Agency, CIA, FBI) (now turn our attention to testing for counterproductive behaviors; spies and spooks) 45

Frequency of false positives; that is, there is a high degree of error with respect to finding that an individual is lying when in fact, the individual is not (details below, NFE) Assume 90 percent accuracy (high end estimate) Assume rate of stealing is 5% of the working population If 1, 000 polygraphs were given, we would expect 50 individuals would be lying, and given 90% accuracy, 45 of those would be detected However, the problem lies with the other 950 individuals 95 (950 X. 10) would be identified as lying when they had not Thus, 140 individuals would be identified as having lied, with 68% of them being false positives Not good (text actually gives 3, I am asking you to learn the major one; (a) other reactions than guilt can trigger an emotional response; (b) there are countermeasures that can be used to avoid detection - I am sure you can find them on the web) 46

There are two basic types of paper and pencil integrity tests Overt integrity tests ▪ Self-report inventories that measure a job applicant’s “attitudes” and “cognitions” toward theft that might predispose him/her to steal at work Personality-based measures ▪ Self-report inventories that measure integrity as part of a larger syndrome of antisocial behavior or organizational delinquency and thus not only measure theft but things like drug and alcohol abuse, vandalism, sabotage, assaultive actions, insubordination, absenteeism, excessive grievances, bogus worker compensation claims and violence (this slide NFE; many states ban these again due to high false positives; read questions from 617; read 618, 1) 47

Pencil and paper integrity tests were developed to replace polygraph testing after the Employee Polygraph Protection Act was passed in 1988 A few states have passed laws against the use of these tests as well, so be careful and check the state laws Once again the reason for concern is the high number of false positives that occur Because of the concern about theft by employers, the use of integrity tests is on the rise and thus more validity studies have been conducted recently (this slide NFE) 48

29 B FE: These tests do correlate with measures of general counterproductive behaviors (theft measures, grievances filed, absenteeism, disciplinary actions, etc. ), but it is not clear yet the extent to which they predict/are related to job performance (NFE) They appear to be OK to use in a selection program, as long as it in states where they are legal, however, at the current time, many still oppose their use False positives and the social implications of that – how would you like to be identified as a liar and cheater when you were not? Frequency of false positives is unknown 49

These tests do not have adverse impact and thus are not likely to result in legal complaints on that basis, but ADA does prohibit items relating to previous drug and alcohol use/abuse Thus, if there any such items on the test, organizations are in legal jeopardy These tests appear to be OK to use in a selection program, as long as it in states where they are legal, however, at the current time, many still oppose their use False positives and the social implications of that – how would you like to be identified as a liar and cheater when you were not? Frequency of false positives is unknown (given the caveat above) 50

Use of drugs and alcohol have been a major concern since the 1960 s (casting dispersions on my generation, the “hippie” generation) Research clearly indicates that drug use is negatively associated with job performance and positively correlated with accidents, injuries, absences, involuntary turnover (firing), and job-withdrawal behaviors such as taking too many breaks and sleeping on the job Prevalence data are dated but somewhat startling (moving onto drug testing: two slides on prevalence) 51

Recent data do indicate that drug use is increasing Increases in use of marijuana have been as high as 43% in states that have legalized marijuana If you have been watching the news at all in the past couple of years, use of prescription opiates is considered to have reached an epidemic level Thus, I think we can “safely” assume that drug use in the workplace is increasing (two slides on prevalence) 52

Hard to believe, however, that people would answer paper and pencil test questions honestly ▪ Do you think that it is OK for workers to use “soft” drugs at work if this does not cause poor performance? ▪ In the past six months, how often have you used marijuana at work? ▪ In the past six months, have you brought cocaine to work even though you did not use it at work? SO 31 B: No public studies that have evaluated either the reliability or validity of these tests So, bottom line is that we really have no clue about these tests (I find it hard to believe anyone would answer these types of questions honestly!) 53

In one court case, the court ruled that these were illegal based on the Fifth Amendment’s prohibition against involuntary self-incrimination (Have to be careful: remember even marijuana use is illegal at the federal level) 54

The legal status of drug testing is unclear Organizations face less risk using drug testing for preemployment selection (testing individuals who are applying for a job) They face considerably more risk if they test existing workers for promotions (or transfers) or testing workers to detect drug users for disciplinary or counseling purposes NFE, but why? Applicants cannot take advantage of collective bargaining or challenge employment at-will principles, as can employees who feel they have been wrongly treated NFE, Consult a good lawyer in employment law before implementing drug testing for selection purposes! (future? ) 55

To date there haven’t been many cases of genetic discrimination but there have been a few and it’s important to keep watch on that The authors end the chapter with a discussion of neuroscience-based discrimination – again, it’s not here yet, but it could well be here soon…. (last slide – time for EAS? ? ) 56

Questions? ? ? The END 57