Chapter Six Training Evaluation Objectives O Explain why

Chapter Six Training Evaluation

Objectives O Explain why evaluation is important O Identify and choose outcomes to evaluate a training program O Discuss the process used to plan and implement a good training evaluation O Discuss the strengths and weaknesses of different evaluation designs Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Objectives O Choose the appropriate evaluation design based on the characteristics of the company and the importance and purpose of the training O Conduct a cost-benefit analysis for a training program O Explain the role of workforce analytics and dashboards in determining the value of training practices Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Key Terms O Training effectiveness: benefits derived training O Training evaluation: process of determining training effectiveness O Training outcomes: measures to evaluate training effectiveness O Evaluation design: data will be collected and which data will be collected for training evaluation Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Formative Evaluation O Refers to evaluation of training that takes place during program design O Helps to ensure training is well organized, runs smoothly, and that trainees learn and are satisfied O Provides information on how to make a program better Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Summative Evaluation O Refers to evaluation conducted to determine if training has lead to desirable outcomes O Examines whether trainees have improved or acquired knowledge, skills, attitudes, behaviors, or other outcomes O Includes examining the business impact of training Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Importance of Evaluation O There are multiple reasons to evaluate: o identify a program’s strengths and o o weaknesses assess what features of training content and context matter identify which trainees benefited gather information for marketing training determine financial benefits and costs Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

The Evaluation Process Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Training Outcomes Reactions Learning & Cognitive Outcomes Behavior & Skill -Based Outcomes Affective Outcomes Results Return on Investment Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Reactions O Trainees’ perceptions of the training experience relating to the content, facilities, trainer, and methods O Key questions to consider include: o Did the trainees like the program? o Did the environment help learning? o Was the material meaningful? O Typically measured at the end of training Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Learning & Cognitive Outcomes O Relate to familiarity with information, including principles, facts, techniques, procedures, and processes O Typically measured via paper-and-pencil tests and self-assessments O Tests often preferred over self-assessments Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Behavior & Skill-Based Outcomes O Relate to proficiency with technical skills, motor skills, and behavior O Include learning and transfer O Learning often assessed via work samples O Transfer often assessed via observation or managerial/peer ratings Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Affective Outcomes O Include attitudes and motivation o e. g. , self-efficacy, employee engagement, motivation to learn, tolerance for diversity, attitudes toward safety O The attitude of interest depends on the training objectives O Affective outcomes often measured via surveys Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Results O Did training have an impact on meaningful business outcomes? O Outcomes used to determine the benefits of training to the company o e. g. , reduced costs, increased employee retention, increased sales, improved quality or customer service Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Return on Investment O Involves comparing the training program's benefits to its costs in monetary terms o Benefits: value the company gains o Direct costs: salaries for employees involved in training, program materials, facilities, and travel o Indirect costs: costs not related directly to design and delivery Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Which Outcomes? Relevance Reliability Discrimination Practicality Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Relevance O The extent to which training outcomes are related to the learned capabilities emphasized in the program O Contamination refers to the inclusion of inappropriate or irrelevant outcomes O Deficiency refers to the omission of important information Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Deficiency, Relevance, & Contamination Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Reliability O The extent to which outcomes can be measured consistently over time O Evaluators are concerned with consistency over time, such that items do not change in meaning or interpretation over time Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Discrimination O The extent to which measured performance reflects a true difference O We want tests that discriminate between high and low performance o A test that is too easy may not discriminate o In this instance, both high and low performers would do well and appear “good” even though they are not Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Practicality O The extent to which outcomes can be easily measured and collected O Companies often claim that measurement is too burdensome Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Training Evaluation Practices Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Which Outcomes? O While collecting data on all outcomes is ideal, it may not always be necessary O Consider the scope of the training and practical considerations O Do not assume that positive reactions lead to transfer Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Threats to Validity O Threats to validity refer to factors that will lead an evaluator to question the results O Internal validity is the believability of the study O External validity is the generalizability of the evaluation results to other groups and situations Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Controlling for Threats O There are three ways to control for threats: o Use pre-tests and post-tests o Use a control group o Random assignment of employees to control and training groups Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Post-test Only O Involves collecting only post-training outcomes O Appropriate when trainees can be expected to have similar levels of proficiency prior to training O Strengthened with a control group Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Pre-test / Post-test O Involves collecting both pre-training and post-training outcomes to determine if a change has occurred O Is there a difference in outcomes before and after training? Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Pre-test / Post-test with Comparison Group O Includes pre-training and post-training outcomes and use of a control group O If the post-training improvement is greater for the training group, there is evidence that training was responsible Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Time Series O Involves collecting measures at periodic intervals pre- and post-training O A comparison group may be used O The strength of this design can be improved by using reversal, which refers to a time period when participants no longer receive training Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Solomon Four-Group O Combines the pre-test/post-test comparison group design and the posttest -only control group design o Pre-test, treatment, post-test o Pre-test, no treatment, post-test o No pre-test, no treatment, post-test Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

When Evaluation May Not Be Necessary O Time constraints O Managers and trainees may lack expertise O The company may view training as an investment from which it expects little or no return Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

When Evaluation is a Must O The training is ongoing and has the potential to affect many employees O The training program involves multiple classes and a large number of trainees O The expertise exists to evaluate O The cost of training is significant O There is sufficient time and interest Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Cost-Benefit Analysis O Companies may desire to quantify whether the benefits of training outweigh the costs O Cost-benefit analysis o Process of determining the economic impact of training using accounting methods that look at training costs and benefits Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Cost Categories 1. 2. 3. 4. 5. 6. 7. program development or purchase instructional materials equipment and hardware facilities travel and lodging salary of the trainer and support staff cost of lost productivity or replacement workers while trainees are away Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Determining Benefits O A number of methods can be used to identify benefits o literature that summarizes benefits o pilot training programs o observing successful job performers asking trainees and managers for estimates Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Calculating Return on Investment (ROI) O O O Identify annual change in outcomes Place a monetary value on the outcomes Determine the annual change in value Determine training costs Calculate net benefit Divide net benefits by costs Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Utility Analysis O Assessing the dollar value of training based on: o estimates of the difference in performance between trained and untrained employees o number of individuals trained o length of time training is expected to influence performance o variability in performance in the untrained group of employees Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Success Cases & Return on Expectations O Success cases refer to concrete examples showing how learning has led to results the company finds worthwhile and credible O Return on expectations (ROE) demonstrates to key stakeholders that their expectations about training have been satisfied Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Big Data O Big data refer to complex datasets compiled across different systems, including marketing, sales, HR, finance, accounting, customer service, and operations O Three dimensions characterize big data o volume o variety o velocity Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.

Big Data & Training O Big data help make decisions about human capital based on data, rather than intuition and conventional wisdom O Big data can be used to: o evaluate the effectiveness of programs o determine their impact on business results o develop predictive models forecasting training needs, course enrollments, and outcomes Copyright © 2017 Mc. Graw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of Mc. Graw-Hill Education.