Fire Police Selection Inc Dan Biddle Ph D

Fire & Police Selection, Inc. Dan Biddle, Ph. D. Stacy L. Bell, M. S. August 26, 2011 A National Fire Chief Survey Reveals the Key Ingredients to Include in Your Fire Department Test

Contact Information Dan A. Biddle, Ph. D. CEO, Biddle Consulting Group, Inc. /Fire & Police Selection, Inc. Stacy L. Bell, M. S. Executive Vice President, Fire & Police Selection, Inc. 193 Blue Ravine, Ste. 270 Folsom, CA 95630 1 -888 -990 -3473 www. FPSI. com | www. biddle. com Email: dan@biddle. com | stacy@FPSI. com Copyright © 2011 Fire & Police Selection, Inc.

Biddle Consulting Group, Inc. • Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense) • EEO Litigation Support / OFCCP (federal contracting) Audit Support • Compensation Analyses / Test Development and Validation • Published: Adverse Impact and Test Validation, 2 nd Ed. , as a practical guide for HR professionals • Editor & Publisher: EEO Insight an industry e-Journal • Creator and publisher of a variety of productivity Software/Web Tools: – OPAC® (Administrative Skills Testing) – Criti. Call® (9 -1 -1 Dispatcher Testing) – Auto. AAP™ (Affirmative Action Software and Services) Industry – C 4™ (Contact Center Employee Testing) Leader – Encounter™ (Video Situational Judgment Test) – Adverse Impact Toolkit™ (free online at www. disparateimpact. com) – Auto. GOJA® (Automated Guidelines Oriented Job Analysis®) Copyright © 2011 Fire & Police Selection, Inc.

Fire & Police Selection, Inc. • • Over 80 Physical Ability Test installations at fire departments across the U. S. Over 800 personnel and fire and police clients across the U. S. and Canada Entry-level written, physical, interview assessments, manipulative skills tests, cognitive ability tests, and personality tests available for selection of public safety personnel Post-Academy tests Fire Promotional tests and assessment exercises Online applicant testing Custom job analysis and validation services Copyright © 2011 Fire & Police Selection, Inc.

Presentation Overview • • • Personnel Testing Overview What is Validation and Why Do I Need It? Test Usage The National Fire Select Test—A Case Study Benefits to a Balanced Testing Process Copyright © 2011 Fire & Police Selection, Inc.

Approach to Test Validation The Quickly Changing Climate of Personnel Testing… • EEOC • DOJ • DOL (OFCCP—Office of Federal Contract Compliance Programs) • State EEO Offices of Fair Employment • Private Plaintiff Groups Copyright © 2011 Fire & Police Selection, Inc.

Adverse Impact & Test Validation Uniform Guidelines & Civil Rights Act of 1991 Amends Section 703 of the 1964 Civil Rights Act (Title VII) (k)(1)(A). An unlawful employment practice based on disparate impact is established under this title only if: A(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and consistent with business necessity; OR, • A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate employment practice, and the respondent refuses to adopt such alternative employment practice. • Copyright © 2011 Fire & Police Selection, Inc.

How Can Testing Practices be Challenged? Title VII Disparate Impact Discrimination Flowchart Practice, Procedure, or Test (PPT) Adverse Impact? “OR” YES END Is the PPT Valid? YES Alternative Employment Practice? NO Defendant Prevails YES Plaintiff Prevails NO NO Plaintiff Prevails Copyright © 2011 Fire & Police Selection, Inc.

Validation Standards Court Precedence • Title VII: Uniform Guidelines on Employee Selection Procedures (1978): EEOC, DOJ, CSB, DOL • Principles for the Validation and Use of Personnel Selection Procedures (2003) • • Division 14 of the American Psychological Association (the Society for Industrial and Organizational Psychology, or SIOP) • Standards for Educational and Psychological Testing, published by: • • • American Educational Research Association American Psychological Association, and National Council on Measurement in Education Copyright © 2011 Fire & Police Selection, Inc.

What is Validation? Traditionally, it is making sure a selection procedure (i. e. , test) measures what it is designed to measure • In a legal realm, a selection procedure is valid if it can be proven by an employer that it is “…job related for the position in question and consistent with business necessity. ” • Copyright © 2011 Fire & Police Selection, Inc.

Great Job Performance Copyright © 2011 Fire & Police Selection, Inc.

Reliable… but not Valid Consistently hiring the wrong people Copyright © 2011 Fire & Police Selection, Inc.

Neither Reliable or Valid Hit or Miss… Copyright © 2011 Fire & Police Selection, Inc.

Reliable and Valid! Reduced Turnover & Absenteeism Better Productivity More Satisfied Clients Less Time to Train Employee Satisfaction Increased Less Liability Copyright © 2011 Fire & Police Selection, Inc.

Uniform Guidelines Style Validation… Content Validity • Section 14 C • Criterion Validity • Section 14 B • Construct Validity • Section 14 D • Copyright © 2011 Fire & Police Selection, Inc.

Content Validity Essential #3: Connecting the Test to the Job Duties Other KSAOs Operationally defined KSAOs Selection Devices (e. g. , application form, tests, interviews) Copyright © 2011 Fire & Police Selection, Inc. Content Valid!

Test Usage Pass/Fail Cutoffs: – “Normal Expectations of Acceptable Proficiency in the Workplace” (Guidelines, 5 H) – Modified Angoff (U. S. v. South Carolina, USSC) n Banding: – Substantially Equally Qualified Applicants – Statistically Driven (use Std. Error of Difference) n Ranking: n – Is there adequate score dispersion? – Does the test have high reliability? (e. g. , >. 85) – Is the KSA performance differentiating? n Weighted/combined with other tests – How are the weights related to the job – Do they come from the job analysis or SME ratings? Copyright © 2011 Fire & Police Selection, Inc.

Using Test Scores • Ranking assumes one applicant is reliably more qualified than the other • Banding considers the unreliability of the test battery and “ties” applicants • Pass/fail cutoffs treat all applicants as either “qualified” or “not qualified” Fire & Police Selection, Inc. 2007 www. fpsi. com 18

Pass/Fail Cutoffs • Some major considerations: • NOT ADVISED: • Using an arbitrary cutoff (e. g. , 70%) • Using applicant scores to benchmark (e. g. , setting cutoff scores at mean-SD of applicant scores) • ADVISED: • Consider “Normal expectations of acceptable proficiency in the workplace” (Guidelines, 5 H) • Usually requires SME-level data or ratings • # Applicants that can be feasibly processed • CONSIDER: • Is the test supported by content validity information or criterionrelated information? • How critical are the KSAs measured? • Does the test measure “baseline” or “differentiating” KSAs? • How would current incumbents perform on this test? Copyright © 2011 Fire & Police Selection, Inc.

Comparison Between “Test Use” Options Factor Ranking Banding Validation Requirements High Moderate Low Adverse Impact High Moderate Low Defensibility Low High Litigation "Red Flag" High Moderate Low Utility High Moderate Low Cost Low Moderate High Applicant Flow Restrictive/Controllable Moderate/Controllable High Development Time Low Moderate High Reliability Requirements High Moderate Low # Item Requirements High Moderate Low Copyright © 2011 Fire & Police Selection, Inc. Pass/Fail Cutoffs

The Value of a Standardized Written Test • Candidates bounce from one department to the next taking entry-level firefighter tests that measure different skills and abilities when in fact the job requirements rarely differ from department to department. • There is value to a standardized national firefighter written test similar to the concept of the standardized physical ability test (e. g. , C-PAT). • How can we measure critical skills and abilities that are standard entry-level requirements across fire departments in a way that is job-related and fair to candidates? Copyright © 2011 Fire & Police Selection, Inc.

Selecting the Ideal Firefighter Mechanical Aptitudes Critical Skills & Abilities Test Map “A” Reading Test “B” Reading Ability Mathematical Ability Writing Ability Interpersonal Skills NFST™ Decision“C” Making Test Spatial Abilities Copyright © 2011 Fire & Police Selection, Inc.

Development & Validation of the National Fire Select Test™ • Leveraged cumulative firefighter job analysis data from the last decade to identify which skills and abilities are consistently rated as being “critical to successful firefighter job performance. ” • Administered the “Buy the Perfect Firefighter” survey to build the test content and test weights. • Used Subject-Matter Experts (SMEs) and job performance ratings to identify a job-related cutoff. Copyright © 2011 Fire & Police Selection, Inc.

Development & Validation of the National Fire Select Test™ • FPSI sent surveys to over 1, 000 chief officers from fire departments across the U. S. who were asked to weigh the importance of key critical skills and abilities required for successful entry-level firefighter job performance. • Respondents were asked to evaluate both cognitive (e. g. , “Academic Success”) and non-cognitive (e. g. , “Personal Characteristics”) skills and abilities. Copyright © 2011 Fire & Police Selection, Inc.

National Survey—Part I Buy the “Perfect Firefighter” There are two parts to this survey. The first section asks you to spend $100. 00 to “buy” the perfect Entry-Level Firefighter by spending this money within two major categories: Cognitive/Academic and Personal Characteristics. The second section asks you to spend $100 within both of these categories. • Section One: Distribute $100 Between Two Qualification Areas Cognitive/Academic Skills/Abilities (see detailed list below) ____% Personal Characteristics (see detailed list below) ____% Copyright © 2011 Fire & Police Selection, Inc.

National Survey—Part II Cognitive/Academic Skill/Ability $ to Spend Description Math Basic mathematics to add, subtract, multiply, divide. Reading Ability to read, retain, recall, and comprehend written material. Verbal Communication Verbally communicate in English. Writing ability including spelling, grammar, sentence structure. Map Reading Ability to read maps and mapped information. Problem-Solving Problem solving and analysis Strategic Decision-Making Decision-making ability to carry out assignments. Mechanical Ability Sufficient to operate, use, and/or repair equipment. Total must sum to $100 Copyright © 2011 Fire & Police Selection, Inc.

National Survey—Part II Personal Characteristics Personal Characteristic $ to Spend Description Teamwork Sufficient to work and live harmoniously with co-workers and to consistently follow-through on routine duties Working Under Stress Ability to perform duties under stressful emergency situations. Allegiance/Loyalty Have allegiance to department’s core values and mission. Truthfulness/Integrity demonstrate truthfulness/integrity to self and others even in the face of severe consequences. Public Relations Willing to be resourceful, goal-oriented, and proactive in bringing tasks to completion. Emotional Stability Willing to remain calm under pressure or heavy workloads, not easily given to hostility, anxiousness, or vulnerability. Sensitivity Ability to maintain respect for other’s emotional needs following a crisis. Proactive/Goal-Oriented Represent public relations and public perception of department’s image. Thoroughness/Attention to Detail Thoroughness and attention to detail in performing all duties. Following Orders Receives and responds to orders or instructions from superiors. Total must sum to $100 Copyright © 2011 Fire & Police Selection, Inc.

U. S. Fire Department Survey Results National Fire Select Test™ Personal Characteristics 55% Cognitive/Academic Skills & Abilities 45% Personal Characteristics Component Cognitive/Academic Component Mechanical Ability 14% Following Orders 11% Mathematical Reasoning 15% Strategic Decision. Making 14% Teamwork 12% Thoroughness/ Attention To Detail 9% Reading Ability 18% Working Under Stress 10% Public Relations 9% Allegiance/ Loyalty 10% Sensitivity 8% Problem Solving 14% Writing Ability 15% Map Reading 10% Emotional Stability 10% Copyright © 2011 Fire & Police Selection, Inc. Proactive/ Goal-Oriented 8% Truthfulness/ Integrity 13%

U. S. Fire Department Survey Results Test Section # of Items Cognitive/ Academic Skills Personal Characteristics 55 Weighted Value 45% 26 43% Work Styles Inventory 85 12% Copyright © 2011 Fire & Police Selection, Inc.

Cutoff Score Determination § § 12 Fire Chiefs reviewed the academic (cognitive) test items and were asked to identify the difficulty level. The difficulty levels of the remaining cognitive and non-cognitive test items, which were validated using criterion-related validity, were calculated from the original validation studies. These score bands were derived using the composite Standard Error of Difference (SED) from the validation sample scores (N = 728). The composite SED is calculated by taking the composite SEM and multiplying it by the square root of 2. Band A was calculated by taking the highest score received on the NFST validation administration and reducing it by one composite SED. Band B was calculated by taking the next highest score below the lowest score in Band A and reducing it by one composite SED. This process was continued until the composite adjusted critical score was reached (e. g. , a 70. 704% on the test). The SED was used to create these score bands because it effectively provides meaningful score breaks that represent different levels of the attributes measured. If ranking is used for the NFST, FPSI recommends utilizing the statistically derived bands shown in Table 1 (rather than top-down ranking). Band Highest Score Lowest Score A 100% 86. 246% B 86. 245% 80. 297% C 80. 296% 74. 336 D 74. 335% 70. 704% Copyright © 2011 Fire & Police Selection, Inc.

Failure to Use a “Balanced Test” • • • No validity Academy failure rates increase and/or job performance suffers Less minority representation (i. e. , adverse impact) in the final eligibility list Might hire mostly “book smart” but not “people smart” firefighters Greater potential for lawsuits which almost always results in lowering standards Copyright © 2011 Fire & Police Selection, Inc.

Fire & Police Selection, Inc. Contact us at: Mailing Address • 193 Blue Ravine Rd. Suite 270 • Folsom, CA. 95630 • Website • www. FPSI. com • www. National. Fire. Select. com • • Telephone • 916. 294. 4242 x. 245 • E-Mail • stacy@fpsi. com Copyright © 2011 Fire & Police Selection, Inc.