Creating an Aligned Assessment System to Promote Work

Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice President, Research ACT, Inc. Though designed to meet a wide array of needs, all ACT programs and services have one guiding purpose: Helping people achieve education and workplace success. © 2010 by ACT, Inc. All rights reserved. 1

The Challenges • Create a comprehensive assessment system that predicts success in education and work • Tailor assessments to critical transition points and context • Understand the complexity and interplay of cognitive and noncognitive factors across time and jobs © 2010 by ACT, Inc. All rights reserved. 2

Do noncognitive factors matter? • What we know from the world of work: – Project A from the military – Training and work outcomes © 2010 by ACT, Inc. All rights reserved. 3

Combining Personality & Cognitive Ability Tests Creating opportunity for incremental validity especially as criteria vary Correlations between general cognitive ability and personality tests and measures of job performance in Project A Cog Pers Both Criteria . 63 . 26 . 67 Core technical proficiency . 65 . 25 . 70 General Soldiering Proficiency . 31 . 33 . 44 Effort and Leadership . 16 . 32 . 37 Personal Discipline . 20 . 37 . 42 Physical fitness and military bearing (Mc. Henry, Hough, Toquam, Hanson, & Ashworth, 1990) © 2010 by ACT, Inc. All rights reserved. 4

Training and Work Outcomes • Noncognitive factors are predictive of important job performance criteria (e. g. , Rotundo & Sackett, 2002; Casillas et al. , 2009) – – Task Performance Organizational Citizenship Counterproductive Work Behaviors Safety • Noncognitive factors meet a variety of employer needs – Screening – Selection – Training © 2010 by ACT, Inc. All rights reserved. 5

Validity of Cognitive and Noncognitive Measures for Training and Work Outcomes Training Outcomes Work Performance General Mental Ability . 54 a . 62 b Math . 48 a . 52 b Reading . 44 a . 35 b Conscientiousness . 17 c . 22 c Emotional Stability . 10 c . 11 c Notes. a Brown, Le, & Schmidt (2006). b Salgado et al. (2003). c Schmidt et al. (2007) corrected for indirect range restriction. © 2010 by ACT, Inc. All rights reserved. 6

No Unified Theory of Key Characteristics • However, the literature suggests that several factors are important: – Motivation, Social Engagement, and Self-Regulation – Personality characteristics (the Big Five) • Role of behavior ratings vs. self-report – Different perspectives – Coaching/training – Monitoring progress © 2010 by ACT, Inc. All rights reserved. 7

Common and Specific Factors • Are the same characteristics (i. e. , common factors) associated with educational and work outcomes? • Are there unique (i. e. , specific) characteristics that contribute to the prediction of educational and work outcomes? • In particular, how do age and setting impact the outcomes of interest? © 2010 by ACT, Inc. All rights reserved. 8

Integrated Pyramid for Success So cia l Ca pit al Career Development (Exploration, Crystallization, Choice & Match) rk o l/W s o ho ctor c S Fa Behavioral Development (Motivation, Social Engagement, & Self-Regulation) Cognitive Development & Acquisition of Foundational Skills (Academic Learning & Achievement) Allen & Robbins (2010) © 2010 by ACT, Inc. All rights reserved. 9

The human side of the pyramid © 2010 by ACT, Inc. All rights reserved. 10

Set of Solutions ACT’s noncognitive programs are part of a comprehensive and integrated set of solutions, which address the three broad areas essential for academic and work success across the lifespan: § Academic achievement & foundational skills § Behavior (noncognitive) § Career planning (noncognitive) © 2010 by ACT, Inc. All rights reserved. 11

Comprehensive & Integrated Academic & Work Solutions: Grade 6 - Work 6 Achievement & Foundational Skills Behavior Career Planning 7 Grades 8 9 10 11 12 EXPLORE® PLAN® ACT® College Work COMPASS® Work. Keys® SRI and BMS ACT Interest Inventory SRI Performance, Talent & BMS College Fit © 2010 by ACT, Inc. All rights reserved. 12

Creating a Comprehensive Set of Behavior (Self & Other Report) Assessments • Role of meta-analysis and validity generalization – Robbins et al. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261 -288. – Robbins et al. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated metaanalytic path analyses. Journal of Applied Psychology, 94, 1163 -1184. © 2010 by ACT, Inc. All rights reserved. 13

Behavior: Self-Report Domain Middle School High School & College Work* Motivation (Getting work done) § Academic Discipline § Commitment to School § Optimism § § § Social Engagement (Getting along) § Family Attitude toward Education § Family Involvement § Relationships w/ School Personnel § School Safety Climate § Social Activity § Social Connection § § § Self-Regulation (Keeping your cool) § Managing Feelings § Thinking Before Acting § Orderly Conduct § Academic Self-Confidence § Steadiness § Optimism § Stability Academic Discipline Commitment to College Goal Striving General Determination Study Skills Communication Skills § Carefulness § Discipline § Order Cooperation Goodwill Influence Sociability Striving Note. *Creativity and Savvy do not fully map onto these domains. © 2010 by ACT, Inc. All rights reserved. 14

Behavior: Other Report Domain Middle School & High School Work Motivation (Getting work done) § § Social Engagement (Getting along) § Communication § Working with Others Self-Regulation (Keeping your cool) § Managing Feelings § Conduct § § Initiative Planning & Organizing Sustained Effort Performance § § Initiative Planning & Organizing Persistence Responsibility Organizational Citizenship Stress Management Following Rules Adaptability © 2010 by ACT, Inc. All rights reserved. 15

Career Planning Domain Middle School & High School Work Interests § ACT Interest Inventory § DISCOVER Career Information Decision System § Fit Inventory Values § DISCOVER Career Information Decision System § Fit Inventory - Neuman et al. (2009). Job congruence, academic achievement, and earnings - Tracey & Robbins (2006). The interest-major congruence and college success relation: A longitudinal study. © 2010 by ACT, Inc. All rights reserved. 16

Tailoring Assessments to Critical Transition Points and Context • The role of noncognitive measures in promoting work training readiness and career success – Talent supply chain – Meeting various needs © 2010 by ACT, Inc. All rights reserved. 17

Talent Supply Chain Pre-K Primary (K-8) Secondary (9 -12) Post Secondary (2 & 4 -yr) Work • Noncognitive factors are important at all of the key transitions that lead individuals to the world of work. © 2010 by ACT, Inc. All rights reserved. 18

Noncognitive Measures Meet Various Workforce Needs Purpose Needs ACT Noncognitive Solutions Pre-Selection -- Screen people in most cost effective way Work. Keys Performance -- Find honest/dependable employees -- Save time in the screening process Recruitment -- Identify people who fit the work Work. Keys Fit environment -- Identify people with skills that match the job Selection -- Select employees with skills that best fit the job -- Save time in selection process -- Select people in most cost-effective way -- Find honest/dependable employees -- Certifying employees Work. Keys Performance Work. Keys Talent Work. Keys Fit © 2010 by ACT, Inc. All rights reserved. 19

Noncognitive Measures Meet Various Workforce Needs (cont. ) Purpose Needs ACT Noncognitive Solutions Coaching/ Development -- Identify other jobs that an employee can fit -- Develop employees for future company needs -- Employees identify areas of improvement Work. Keys Talent Work. Keys Fit Behavioral Monitoring Scales Succession/ Leadership Planning -- Identify candidates for top-level positions -- Develop employees for future needs of company -- Retain top performers Work. Keys Talent Work. Keys Fit Training/ Development -- Identifying work readiness -- Identify basic workplace skill levels -- Educating about career planning -- Job Placement Work. Keys Talent Work. Keys Fit Behavioral Monitoring Scales © 2010 by ACT, Inc. All rights reserved. 20

Critical Issues • Self vs. other report • Changing attitudes, behaviors, and skills • Complexity of personality & career relationships – Different personality characteristics are important for different jobs (whether across or within jobs) – Moderation effects of personality on ability – Curvilinear effects of personality on performance © 2010 by ACT, Inc. All rights reserved. 21

Self vs. Other Report • Connelly & Ones (2010) conducted meta-analyses of self-other ratings based on 44, 000+ individuals across 263 independent samples. – Other ratings are strong predictors of behaviors, particularly for academic achievement and job performance. – In some cases, other ratings yielded predictive validities substantially greater than self-ratings. – When other-ratings are added to self-ratings, results show considerable increases in validity (with gains more pronounced for Conscientiousness, Emotional Stability, and Extraversion). © 2010 by ACT, Inc. All rights reserved. 22

Changing Attitudes, Behaviors, and Skills Behavior modeling training (BMT), based on Bandura’s social learning theory, has become a widely used psychologically-based approach to training in work environments. A recent meta-analysis by Taylor et al. (2005) summarizing 117 studies found substantial effects of BMT on a variety of training outcomes. Outcome d Attitudes . 56 Behaviors . 33 Declarative knowledge . 69 Procedural knowledge-skills 1. 21 © 2010 by ACT, Inc. All rights reserved. 23

“It was about here, wasn’t it, Ed, when you came on board as sales manager? ” Harvard Business Review. March 2007. p. 90 © 2010 by ACT, Inc. All rights reserved. 24

Complexity of Personality & Career • Different personality characteristics are important for different jobs (whether across or within fields). – Conscientiousness appears to be important for all jobs. – Extraversion is relevant to some jobs (e. g. , sales) but not others (e. g. , computer programmers). © 2010 by ACT, Inc. All rights reserved. 25

Complexity of Personality & Career Across Fields • Position Computer Programmers • O-NET Job Zone Four: Considerable Preparation Needed • Education Most occupations in this zone require a four-year bachelor's degree, but some do not. http: //online. onetcenter. org/link/summary/15 -1021. 00 • Position Sales Representatives • O-NET Job Zone Four: Considerable Preparation Needed • Education Most occupations in this zone require a four-year bachelor's degree, but some do not. http: //online. onetcenter. org/link/summary/41 -4011. 00 © 2010 by ACT, Inc. All rights reserved. 26

Complexity of Personality & Career: Comparing Scores Across Fields © 2010 by ACT, Inc. All rights reserved. 27

Complexity of Personality & Career Moderation Postlethwaite et al. (2009) found that conscientiousness was a stronger predictor of safety behavior for individuals with lower levels of cognitive ability. © 2010 by ACT, Inc. All rights reserved. 28

Complexity of Personality & Career Curvilinear Relationships Le et al. (in press) found curvilinear relationships between Emotional Stability and job performance (task, OCB, CWB), as well as between Conscientiousness and task performance. © 2010 by ACT, Inc. All rights reserved. 29

Questions? Correspondence regarding this presentation should be addressed to: Steve Robbins Vice President, Research ACT, Inc. steve. robbins@act. org © 2010 by ACT, Inc. All rights reserved. 30

References Allen, J. & Robbins, S. (2010). Effects of interest–major congruence, motivation, and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23– 35. Brown, K. G. , Le, H. , & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there incremental validity for training performance? International Journal of Selection and Assessment, 14, 87 -100. Casillas, A. , Robbins, S. B. , Mc. Kinniss, T. , Postlethwaite, B. , & Oh, I. S. (2009). Using narrow facets of integrity to predict safety: A test validation study. International Journal of Selection and Assessment, 17, 119 -125. Connelly, B. S. & Ones, D. S. (2010). An Other Perspective on Personality: Meta-Analytic Integration of Observers’ Accuracy and Predictive Validity. Psychological Bulletin, 136, 1092– 1122. Le, H. , Oh, I. -S. , Robbins, S. B. , Ilies, R. , Holland, E. , & Westrick, P. (in press). Too Much of a Good Thing: Curvilinear Relationships Between Personality Traits and Job Performance. Journal of Applied Psychology. Mc. Henry, J. J. , Hough, L. M. , Toquam, J. L. , Hanson, M. A. , & Ashworth, S. (1990). Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, 335 -354. Neumann, G. , Olitsky, N. , & Robbins, S. (2009). Job congruence, academic achievement, and earnings. Labour Economics, 16, 503 -509. Postlethwaite, B. , Robbins, S. , Rickerson, J. , & Mc. Kinniss, T. (2009). The moderation of conscientiousness by cognitive ability when predicting workplace safety behavior. Personality and Individual Differences, 47, 711 -716. © 2010 by ACT, Inc. All rights reserved. 31

References (cont. ) Robbins, S. B. , Lauver, K. , Le, H. , David, D. , Langley, R. , & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261 -288. Robbins, S. B. , Oh, I. , Le, H. , & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, 1163 -1184. Rotundo, M. & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policycapturing approach. Journal of Applied Psychology, 87, 66– 80. Salgado, J. F. , Anderson, N. , Moscoso, S. , Bertua, C. , & de Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European community meta-analysis. Personnel Psychology, 56, 573 -605. Schmidt, F. L. , Shaffer, J. , & Oh. I. (2007). Reassessing the Relative Importance of Cognitive Ability and Personality in Job Performance and Training Performance: Some Surprising New Research Findings. Paper presented at the 2007 ATP conference, Palm Springs, CA. Feb. 6. Taylor, Paul J. , Russ-Eft, D. F. , & Chan, D. W. L. (2005). A Meta-Analytic Review of Behavior Modeling Training. Journal of Applied Psychology, 90, 692 -709. Tracey, T. J. G. , & Robbins, S. B. (2006). The interest-major congruence and college success relation: A longitudinal study. Journal of Vocational Behavior, 69, 64 -89. © 2010 by ACT, Inc. All rights reserved. 32