Professional Development Evaluation Strategies to Maximize Impact Jeffri

Professional Development: Evaluation Strategies to Maximize Impact Jeffri Brookfield National Center for Systemic Improvement & IDEA Data Center Chelsea Guillen Early Intervention Training Program at the University of Illinois 2017 LEADERSHIP CONFERENCE

2 Purposes • To examine a framework for developing an evaluation of improvement strategies that rely on professional development. • To examine the usefulness of information yielded by various measurement tools/approaches. • To examine sample evaluation plans with attention to pivotal decision points in the implementation process. 2017 LEADERSHIP CONFERENCE

3 Importance of Evaluating Professional Development • Professional development (PD) is costly, too often ineffective, and still necessary to establish changes in professional practice. • A good evaluation plan for PD initiatives will: • • Answer the most important questions, In the most useful manner, Allowing for timely corrections/modifications, While minimizing the cost and labor-burden of evaluation. 2017 LEADERSHIP CONFERENCE

Implementing a PD improvement strategy Develop training & select trainees Conduct Training Implement Practice Implementation Support Evaluation Topics • What content? • How did participants • perceive the training? • Who will • Did participants acquire participate? new knowledge/skills? • Did the intervention • Did implementation improve after get implemented as support was offered? planned? What implementation support is needed? 4 2017 LEADERSHIP CONFERENCE

5 Measures: Satisfaction/Perception • How well did the training meet your needs? • Were the trainers knowledgeable? • Did you gain sufficient training in order to…? • What was most valuable (least valuable) about the training? • And so forth…… 2017 LEADERSHIP CONFERENCE

6 Example Satisfaction Survey Satisfaction with a consultant Satisfaction with a training 2017 LEADERSHIP CONFERENCE

7 Measures: Knowledge/Skill • Written assessment • multiple choice, • short answer • Written product • IFSP • Behavioral intervention plan • Assessment report • Other… 2017 LEADERSHIP CONFERENCE

8 Example Knowledge/Skills Measures 2017 LEADERSHIP CONFERENCE

9 Measures: Observational (direct or video) • Rubric • Checklist • Scale • PLACHEK, event, duration, number, frequency, rate…. 2017 LEADERSHIP CONFERENCE

1 0 Sample Observational Measures 2017 LEADERSHIP CONFERENCE

1 1 Participants & Initial Training • Characteristics of participants • Any prerequisites? • Characteristics of the training: • • • Explicit explanation & illustration of practice Opportunities for practice Reflection Sufficient duration and intensity …. . 2017 LEADERSHIP CONFERENCE

1 2 Immediately after training • Knowledge acquired? • Satisfaction with training experience? • Confidence in skills? • Action plan for implementation of new skills? 2017 LEADERSHIP CONFERENCE

1 3 Transfer of training to practice • What practices are being implemented? • Fidelity of implementation? • Frequency/intensity of practice? • Adherence to a standard? 2017 LEADERSHIP CONFERENCE

1 4 Follow-up support • Frequency and quality of feedback? • Support/action plan? • Satisfaction of support experienced? 2017 LEADERSHIP CONFERENCE

Sample measurement tools used by states EITP Quality Professional Development Rubric RI Services Rendered Form (monitoring tool) GVSU Coaching Self-Assessment UIC ERF Observation Coaching Session Summary TACSEI Family Coaching Checklist Implementing the Pyramid Model in Home Visiting Programs: Benchmarks of Quality • Evaluation of Coaching/Support Received • Recommended Practices Performance Checklists • ……. . . • • • 15 2017 LEADERSHIP CONFERENCE

Implementing a PD improvement strategy : Pivotal evaluation points Develop training & select trainees Conduct Training Implement Practice Implementation Support Evaluation Topics • What content? • How did participants • perceive the training? • Who will • Did participants acquire participate? new knowledge/skills? • Did the intervention • Did implementation improve after get implemented as support was offered? planned? What implementation support is needed? 2017 LEADERSHIP CONFERENCE

Planning Guide Implementation Step: __________ 1. Evaluation question(s) [i. e. what do you need to know]? 2. What data will be collected and what tools will be used? 3. When and how will data be collected? 4. When, how, and by whom will data be analyzed? 5. Who will use the results to make decisions, and how and when will these decisions be made? 6. What are potential modifications you may make to the implementation of the improvement strategy based on these data? 17 2017 LEADERSHIP CONFERENCE

1 8 Discussion Questions • Is your state using any of the following? • • Post Training Satisfaction Survey Pre/Post Training Knowledge-Based Assessment Implementation Fidelity Checklist Coaching Satisfaction Survey • What additional measurement tools would be useful to your state for determining how well interventions are being implemented? • What is your plan for evaluating the data being collected? • How will the data be used for continuous improvement? 2017 LEADERSHIP CONFERENCE

Contact information Jeffri Brookfield Chelsea Guillen jbrookf@wested. org cguillen@Illinois. edu https: //ncsi. wested. org/ http: //eitp. education. illinois. edu/ https: //ideadata. org/ 19 2017 LEADERSHIP CONFERENCE

Disclaimer The contents of this presentation were developed, in part, under grants from the U. S. Department of Education, #H 326 R 140006 and #H 373 Y 130002. However, the contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government. Project Officers: Perry Williams and Shedeh Hajghassemali Richelle Davis and Meredith Miceli 2017 LEADERSHIP CONFERENCE