The Kirkpatrick Model evaluating reactions to training Richard

The Kirkpatrick Model evaluating reactions to training Richard Lambert, Ph. D.

Review l How do magic bullets make it into everyday practice? l Teachers must see the need for change. l Barriers to change. l What makes for effective training?

Readiness for Innovation l When is a teacher ready for new and innovative instructional strategies?

Readiness for Innovation l Gotten through the adjustment / survival stage. l Understands how to manage stress. l Has learned behavior management. l Made the commitment to remain in education. l Has learned to be reflective about practice. l Works in a relatively lower stress school.

Review l Mini-process evaluation. l Determine the level of evidence the stakeholders need. l The four levels of evidence.

The Levels The four levels from Kirkpatrick’s model l Level I – Reaction l – How well did the attendees like the training? l Level II – Learning – What concepts, facts, or skills were learned? l Level III – Behavior – What changes in behavior or attitude did the attendees make once back on the job? l Level IV – Results – What were the tangible impacts of the training program on the organization?

The Reaction Level l The typical strategy here is the post- conference questionnaire. l Remember to apply the same good general principles of survey research here that you would to any questionnaire. l Maintain anonymity and confidentiality.

The Reaction Level l Qualitative methods can be very helpful in the item development phase, particularly as they relate to understanding the language that your target audience uses to describe the issues addressed by the survey. l Explore – Investigate - Validate

Survey Format l Avoid ambiguous wording of items. l Structure the survey form so it looks easy to complete. l Structure the survey so it is easy to complete. l Include open-ended questions and an appeal for other comments.

Survey Methods l Pilot-test the survey with three groups: l A small group of potential respondents l Stakeholders – Consumers of the evaluation results l Survey / Research / Evaluation experts

The Pilot Study l Talk through the survey in a focus group format l Record how long it took to fill out l Ask about ambiguous language or confusing wording l Ask what is missing

Survey Content l Determine the survey content based on the needs of the stakeholders. l Use the standard list of item types to discuss with stakeholders the kind of feedback they desire.

The Standard Questions l Did the training session meet your needs / expectations? l Did the content of the training session apply to your work setting? l Rate the mix of lecture and activities used by the trainer. l How well did the trainer state the objectives for the session?

The Standard Questions l Did the trainer keep the session interesting and stimulating? l Did the trainer make effective use of audiovisual / technological aids? l Did the trainer provide effective illustrations / practical examples? l Did the trainer interact with the audience in a friendly and helpful manner?

The Standard Questions l Did the trainer summarize key points effectively? l What benefits did you receive from the training (Knowledge, Skills, Techniques, Attitudes, Beliefs)? l Overall rating of the training session. l Areas for improvement. l Would you recommend the training to others?

Survey Content l Be careful not to include more items than necessary. l Items can be open-ended questions or statements accompanied by a rating scale. l Time constraints and respondent motivation key factors to consider.

Additional Considerations l Follow-up interviews and focus groups can be very helpful supplements to the information obtained from the questionnaire. l Qualitative information can be very useful in forming a complete interpretation of the results of the survey, and will likely elicit valuable feedback beyond what a survey can provide.

Additional Considerations l In some evaluation situations, it may be useful to have both the participants and other stakeholders rate the trainer and the training. l This technique could even include having additional observers attend the training, such as those experienced in training and professional development who possess a basis of comparison from which to make ratings.