PROGNOST A WebBased Personnel Assessment System Cogni Metrics

PROGNOST A Web-Based Personnel Assessment System © Cogni. Metrics, Inc. , 2005

Content navigator PROGNOST: a web-based assessment system • • What is PROGNOST? For whom PROGNOST is designed PROGNOST: Features • • Quality Selection Decisions Local Accessibility User Languages: English and Spanish Expandable with non web-based tests PROGNOST: Applications • • • Selection of Terminal Control Room Operator Selection of Executive Assistent Selection of Retail & Specialty Product Marketing PROGNOST: Specifications • • IT Aptitude Battery Wonderlic Personnel Test Personality Characteristics Inventory Decision Support System Pricing Availability The compass icon appears on many slides in the slide show. Click on one of its arrows to return to the navigator page.

PROGNOST: web-based assessment system What is PROGNOST? PROGNOST a web-based assessment system for personnel selection. PROGNOST consists of: Ø low-cost customizable decision-support system. Ø standard (but expandable) battery of web-delivered tests.

PROGNOST: web-based assessment system For whom PROGNOST is designed for companies that: RETURN • give a high priority to the quality of their selection and promotion decisions. • expect accountability of human resources management. • have employees working in workplaces that are spread out over a region, nation, or internationally. • have employees with ethnic, cultural and linguistic different backgrounds.

PROGNOST features The main goal of our web-based assessment system is to improve the quality of personnel decisions. We will introduce PROGNOST by discussing potential problems with the quality of personnel decisions.

PROGNOST features Quality Selection Decisions Psychological assessment is a systematic effort to collect and to combine relevant information for the purpose of diagnosis or prognosis of human behavior.

PROGNOST features Quality Selection Decisions The quality of psychological assessment often is suboptimal because of the following: Ø Most HR-professionals restrict the systematic effort part to collecting information (i. e. , testing and interviewing). To combine the information experts use so-called “clinical judgment”. Ø Over 50 years of research has shown that combining this information using “clinical judgment” is highly inaccurate and subject to distortion.

PROGNOST features Quality Selection Decisions Ø Until recently no user-friendly system was available to replace “clinical judgment”. Ø Traditional statistical techniques were too restrictive and were not user-friendly.

PROGNOST features Quality Selection Decisions In 2003 Cogni. Metrics, Inc. developed PROGNOST a method to build customized low-cost decision-support systems for selection decisions. A decision support system is an expert system that uses all available scientific knowledge to predict success of current or future employees for a particular job or training, based on available information (such as their test scores). What follows is a short introduction to PROGNOST:

Psychological Assessment RISK ASSESSMENT PER INDIVIDUAL collecting information to predict probability of success in job / training tests questionnaires combining this information into a single statement expressing the probability of success decisionsupport system

s 1 s 2 s 3 s 4 s 5 s 1 s 2 s 3 numbers 4. 66 23. 4 44. 6 similarities s 1 4. 68 s 2 22. 0 s 3 42. 0 INT_factor 4. 70 24. 7 41. 3 23. 3 6. 00 arithmetic s 1 3. 35 s 2 24. 0 s 3 46. 0 comprehension s 1 6. 67 s 2 20. 7 s 3 40. 6 inductive_reasoning s 1 1. 34 s 2 29. 4 s 3 43. 3

s 1 s 2 s 3 s 4 s 5 PROGNOST numbers 4. 66 23. 4 44. 6 23. 3 4. 00 s 1 s 2 s 3 s 4 s 5 similarities 4. 68 22. 0 42. 0 25. 3 6. 00 s 1 s 2 s 3 s 4 s 5 arithmetic 3. 35 24. 0 46. 0 20. 6 5. 99 comp s 1 6 s 2 2 s 3 4 s 4 2 s 5 4 decision support system What do we need it for? Human experts (psychologists, medical doctors) are incapable of reliably combining information of multiple tests into a single decision outcome (e. g. , a hire / no-hire). • • This problem is particularly serious when many information sources (such as tests) have to be considered.

s 1 s 2 s 3 s 4 s 5 PROGNOST numbers 4. 66 23. 4 44. 6 23. 3 4. 00 s 1 s 2 s 3 s 4 s 5 similarities 4. 68 22. 0 42. 0 25. 3 6. 00 s 1 s 2 s 3 s 4 s 5 arithmetic 3. 35 24. 0 46. 0 20. 6 5. 99 comp s 1 6 s 2 2 s 3 4 s 4 2 s 5 4 decision support system How does it work? A decision-support system combines results of scientific studies, common-sense knowledge and observational evidence to come up with a prediction of the probability of success in training or job.

s 1 s 2 s 3 s 4 s 5 numbers 4. 66 23. 4 44. 6 23. 3 4. 00 PROGNOST s 1 s 2 s 3 s 4 s 5 similarities 4. 68 22. 0 42. 0 25. 3 6. 00 s 1 s 2 s 3 s 4 s 5 decision support system How does it work? Theoretical insights Empirical observations arithmetic 3. 35 24. 0 46. 0 20. 6 5. 99 Expert system (BN-model) Common sense PREDICTIONS comp s 1 6 s 2 2 s 3 4 s 4 2 s 5 4

s 1 s 2 s 3 s 4 s 5 PROGNOST numbers 4. 66 23. 4 44. 6 23. 3 4. 00 s 1 s 2 s 3 s 4 s 5 similarities 4. 68 22. 0 42. 0 25. 3 6. 00 s 1 s 2 s 3 s 4 s 5 arithmetic 3. 35 24. 0 46. 0 20. 6 5. 99 comp s 1 6 s 2 2 s 3 4 s 4 2 s 5 4 decision support system How does it work? We build decision support systems using Bayesian Networks, an artificial intelligence technology. Bayesian Networks propagate information derived from objective tests (and subjective ratings if required) to a selection or placement advise for an individual applicant. Bayesian Networks are more powerful and more robust than traditional regression techniques.

s 1 s 2 s 3 s 4 s 5 PROGNOST numbers 4. 66 23. 4 44. 6 23. 3 4. 00 s 1 s 2 s 3 s 4 s 5 similarities 4. 68 22. 0 42. 0 25. 3 6. 00 s 1 s 2 s 3 s 4 s 5 arithmetic 3. 35 24. 0 46. 0 20. 6 5. 99 comp s 1 6 s 2 2 s 3 4 s 4 2 s 5 4 decision support system How does it work? PROGNOST produces the following output: 1. Odds of success: a simple quantitative index. If odds > 1. 00 candidate is likely to succeed on the particular task, else (s)he is more likely to fail. RETURN 2. A more detailed report with the candidates’ strong and weak points.

TESTS collecting information Cogni. Metrics, Inc. offers a standard battery of tests and questionnaires that together predict over 50 % of the variance in training and job performance. • all tests are web delivered • languages: English, Spanish • economical: low assessment costs per candidate • the standard battery can be expanded or tests can be tailored to specific Alcoa needs.

TESTS collecting information The standard battery of PROGNOST administers two main groups of tests / questionnaires: • Intelligence • Personality traits

TESTS intelligence Eighty years of selection research has proven tests of intelligence to be the strongest predictors of success in training and job performance. The standard battery comprises two different types of intelligence tests: • Wonderlic Personnel Test (WPT - Wonderlic, Inc. ) RETURN • IT Aptitude Test (ITAB - Cogni. Metrics, Inc. )

IT Aptitude Battery General: The ITAB tests measure an aspect of intelligence that has become increasingly important since the advent of Information Technology. ITAB tests measure the ability to interact efficiently with systems that have a dynamic of their own. Such systems vary in complexity from MS Office software tools to complex information systems monitoring industrial processes. The cognitive skill required for using such devices is called procedural knowledge, or procedural skill.

IT Aptitude Battery Features: The ITAB test reports deliver: • a multifaceted PERFORMANCE SCORE, measuring Procedural Knowledge Aptitude, and scores on two causative factors: • INTELLIGENCE: efficiency in information processing RETURN • SYSTEMATIC APPROACH: steady approach and careful inferences.

Wonderlic Personnel Test The Wonderlic Personnel Test (WPT) is a classical test of general cognitive ability (general intelligence). It measures primarily so-called crystalyzed intelligence: a product of past learning.

Wonderlic Personnel Test Features: • Short test of general intelligence (test time 12 minutes) with reliabilities and validities comparable with much longer tests. • 50 items in ascending order of difficulty. • Web delivered. RETURN

TESTS personality Certain core personality traits are a predisposition to longterm commitments, such as needed to complete a training program. .

PC Inventory The Personality Characteristics Inventory evaluates candidates on five basic dimensions of personality: • • • Conscientiousness Extraversion Agreeableness Openness Stability

PC Inventory The 150 items can be rearranged in several combinations so as to generate various scores: • 12 subscales: efficiency, dependability, achievement striving, etc. • success scales: commitment to work, integrity, learning, orientation, teamwork. • occupational scales: manager, sales, production worker, driver, clerical employee.

PC Inventory Features: • No time limit (generally takes 25 -30 minutes to complete). • 150 Questions. • Valid, reliable, and legal. RETURN

PROGNOST features Local Accessibility Cogni. Metrics’ tests are web-delivered, which makes test administration possible at virtually any place of the globe. This is ideal for companies that have geographically spread work locations. An additional advantage of our system is that it guarantees an equal weighing of test information throughout the company.

PROGNOST features User languages All tests of the proposed standard battery are available in both English and Spanish.

PROGNOST features Expandable system PROGNOST can be designed to deal with information gathered through observation, interview, and paper-and-pencil tests. For example, the HR specialist may want to add behavior ratings obtained from observing Assessment Center exercises. PROGNOST can be made to accept this information and weigh it appropriately.

Applications examples PROGNOST can be designed to optimally predict any category of employment positions. Cogni. Metrics • reviews empirical studies involving the prediction of success in those positions and estimates the population correlation between predictors and criterion. • translates the outcomes into the structure of the Bayesian Network (conditional probabilities). The following slides show a number current job openings in large San Antonio based companies (Alcoa, Inc. ; H. E. B. , VALERO Energy Corp. ). The purpose is to demonstrate that PROGNOST with the ITAB tests has an edge in predicting success in work environments of the 21 th century.

Applications examples Terminal Control Room Operator (VALERO, Job ID: 5287 -70000393) Position description: Monitors and supports all load rack activities related to the terminal automation system. Monitors terminal automation systems and advise field personnel when a terminal is in need of attention. Tracks product movements to help reduce the chance of terminal outages or possible overfills. Supports troubleshooting of the terminal control system with the assistance of Information Services and/or field personnel. Etc. Experience and Educational requirements: High school diploma or GED required. Some college background preferred. Minimum two years of experience in terminal operations or industry experience required. Basic knowledge of petroleum products and industry terminology required. Excellent PC skills, with proficiency in Microsoft products required. Effective analytical and problemsolving skills. Relevant tests: The ITAB tests measure ability to deal with procedural knowledge, a crucial aptitude in work environments of the 21 th century.

Applications examples Selection of Executive Assistant (ALCOA, Job-ID: 2410 BR) Function description: • Must possess excellent computer skills (Microsoft Word, Excel, Power. Point, Outlook etc. ) • Responsible for travel planning of the Senior Management Team. • The chosen candidate must be able to multi-task, work with confidential information, be self motivated and a team player. • Excellent organizational skills required Experience and Educational Requirements: high school degree or equivalent, bachelors preferred. Requires 10 years of general administrative and secretarial experience with emphasis in working independently towards results and goals. Most relevant tests: ITAB tests measure a crucial aptitude for the 21 th century work environment.

Applications examples Retail & Specialty Product Marketing (VALERO, Job ID: 5359 -22995) Position Description: Performs maintenance and upkeep of Retail facilities. Diagnoses and repairs a variety of refrigeration equipment problems to include HVAC, walk-in coolers, icemakers, and freezers. Diagnoses and repairs a variety of merchandising and fast-food equipment problems such as coffee makers, … cappuccino machines, and related equipment. Maintains and repairs building structures, including roofing, plumbing, drywall, flooring, ceilings, glazing, doors, hardware, electrical, and related building items. Experience and Educational requirements: High school diploma or GED required. Vocational training in craftrelated areas required. Basic computer skills required. Must have experience in selecting proper tools and repair methods. Etc. Relevant tests: Many jobs require technical troubleshooting (diagnoses and repair, see above), a form of procedural knowledge handling. See: ITAB tests.

PROGNOST Specifications PROGNOST consists of: Ø a Bayesian Network Ø a standard battery of tests: § ITAB: IT Aptitude Battery § WPT: Wonderlic Personnel Test § PCI: Personality Characteristic Inventory

Specifications Standard Test Battery IT Aptitude Battery Measurement claim: procedural knowledge handling (aspect of general intelligence) Number of tests: Administration time: Languages: 2 (HTT and BT) 10 minutes per test (average) English, Spanish, German, Dutch Cogni. Metrics, Inc. March 2004 see: www. cognimetrics. com Test Publisher: Published: Price (if purchased separately): Reliabilities: Validities: • • FOR MORE INFO. . . • • construct validity studies: predictive validity studies: Download ITAB manuals: www. cognimetrics. com Report on construct validity: www. internationalmta. org/2004 (paper 066; powerpoint presentation 066). HTT BT (0. 88 – 0. 97) (0. 80 – 0. 87) see: FOR MORE INFORMATION currently under way with: - Air Force Research Lab, Brooks City Base; - Navy Directorate Selection and Classification, Washington, D. C.

Specifications Standard Test Battery Wonderlic Personnel Test Measurement claim: general intelligence Number of tests: Administration time: Languages: Test Publisher: Published: 1 (50 questions) 12 minutes English, Spanish Wonderlic, Inc. 1937 Reliabilities: ranging from 0. 82 – 0. 94 Validities: • construct validity studies: • predictive validity studies: see Manual WPT numerous (see Manual WPT).

Specifications Standard Test Battery Personality Characteristics Inventory Measurement claim: Number of tests: Administration time: Languages: Test Publisher: Published: basic personality dimensions 5 scales, or 12 scales 25 – 30 minutes English, Spanish Wonderlic, Inc. Reliabilities: ranging from 0. 82 – 0. 87 Validities: • construct validity studies: • predictive validity studies: see Manual PCI

Specifications decision support system PROGNOST is a decision-support system build on an expert system shell, a Bayesian Belief Network system. Manufacturer (Belief Network technology and API software): Norsys Software Corp. Vancouver, BC. Canada Network development: Cogni. Metrics, Inc. San Antonio, TX Further software development includes: • Test manager (administering the tests to applicants; transfers test data to database). • User interface decision support system (available to counselors to read decision outcomes and to add further information). • Report writer

Pricing prognost System development Software Development • test manager (incl. interface) $ 22, 500 Network Development $ 6, 500 • user interface decision support system • API programming • report writer These prices hold for the development of a network based on the proposed standard test battery and 5 different job categories to be predicted. Modifications of the standard system, for example, additional tests, and/or additional job categories are possible.

Pricing standard test battery Tests licensing Two options: 1. Quantity-based License: For example, a license for 100 test administrations of the standard test battery costs: $6, 000 + tax. This license can be renewed at any time. Payment in advance. Bulk discounts can be negotiated. 2. Time-limited License: We offer interesting discounts for licenses for a full year of unlimited use of the standard battery. Cogni. Metrics, Inc. can provide an estimate of cost savings resulting from implementation of PROGNOST as basis of your selection process: • direct savings on costs per individual assessment • indirect savings as a result of increased productivity, or improved success ratios in training programs as a result in improved validity of the prediction.

Availability Cogni. Metrics, Inc. can develop the proposed standard system within three months.

Thanks for reviewing this Cogni. Metrics proposal For further information, Please contact: Dr. Martin J. Ippel (210) 481 – 0261 mippel@cognimetrics. com

Cogni. Metrics, Inc. 2004 IT Aptitude Battery: In 2004 Cogni. Metrics, Inc. published the IT Aptitude Battery, the world’s first battery of fully interactive tests of human intelligence. The ITAB tests measure ability to deal with procedural knowledge, a crucial aptitude in work environments of the 21 th century. http: //www. cognimetrics. com The ITAB tests can be purchased as part of PROGNOST, or separately. For more vendor information, please visit: www. cognimetrics. com, select <products>.

Cogni. Metrics, Inc. 2003 PROGNOST: In 2003 Cogni. Metrics, Inc. developed PROGNOST a method to build customized low-cost decision-support systems for selection decisions. A decision support system is an expert system that uses all available scientific knowledge to predict success of current or future employees for a particular job or training, based on available information (such as their test scores). http: //www. cognimetrics. com