FacilitatorEvaluator OrientationBriefing MARC Mass Fatality TTX May 21

Facilitator/Evaluator Orientation/Briefing MARC Mass Fatality TTX May 21, 2013 1

Briefing Topics • Roles and Responsibilities • Evaluation Methodology Overview – Exercise Evaluation Guide (EEG) Concept • Evaluation Process – Evaluator Responsibilities • Assessment, Review, and Analysis of the Exercise 2

Roles and Responsibilities • Facilitators provide situation updates and moderate discussions. They also provide additional information or resolve questions as required. • Evaluators use EEGs to measure and assess performance, capture unresolved issues, and analyze exercise results. Evaluators passively assess and document players’ performance against established emergency plans and exercise evaluation criteria, in accordance with HSEEP standards. Evaluators have a passive role in the exercise and only note the actions/decisions of players without interfering with exercise flow. For this exercise, evaluators may provide a limited coaching role of stimulating thought process. 3

Facilitators • Chris Kates – UC • John Linstrom- MF • Justin Sorg – EM/Logs • Joan Harding– Health and Medical • Dwight Graves PIO • Dana Hall VIC Evaluators • UC – Joe Lynch • MF – Chuck Smith • EM Logs – Ricky Carroll • Health & Medical – Erin Lynch • PIO Dwight Graves • VIC – Lisa Ladue 4

Evaluation Goals • Validate strengths, and identify improvement opportunities • Validate plans, procedures, and protocols • Improve processes and plans • Determine level of capability in regard to the exercise’s core capabilities 5

EEG Concept • The Homeland Security Exercise and Evaluation Program (HSEEP) Volume III provides evaluation and improvement planning policy. • Evaluations should be objective-based to remain in line with: – National Preparedness Goal – Core Capabilities List – Jurisdiction and agency plans • HSEEP EEG templates provide streamlined process for standardized evaluation concept, not necessarily content. 6

EEG Function • Allows evaluators to review validation of plan elements – Plans used will depend on participating agencies • Includes references to plan sections, tables, and charts – Allows evaluators to better prepare for exercise – Provides detail for what will be expected of players • Remains focused on exercise objectives as they relate to the selected core capability 7

Use of EEGs • Evaluation team should use EEGs in conjunction with full plans to prepare • Use EEGs as guides for observation, but do not focus on completing them during the exercise § Use EEGs for data analysis after exercise and in preparation for the evaluator debriefing § Use measured results to make actionable improvement recommendations within the After Action Report (AAR)/ Improvement Plan (IP) 8

Use of Evaluation Tools HSEEP Methodology Steps Evaluation Tools Timeline Exercise objectives, actual plans and procedures Evaluator log EEG analysis sheet AAR data input form Through Final Planning Conference (FPC) planning During exercise Controller/ Evaluator (C/E) Debriefing Post exercise, before AAR draft development 9

Evaluator Notes and Data • Evaluators should capture player discussions, decisions, actions, and activity during exercise play. Use back of EEGs for notes • The EEGs provided should serve as guides for documentation of exercise activity 10

EEG Analysis Form • Each evaluator will be provided with EEGs that will give specific guidance on what data to collect during the exercise, how to record it, and how to analyze it • These forms should be completed at the conclusion of the exercise, using data gathered during the exercise 11

EEG AAR Data Input Form • Each EEG will include a second form to be completed following the C/E Debriefing • This form will serve as an expansion to the notes captured in the EEG analysis form, including an area to provide recommendations. • Evaluators should complete one AAR input form for each identified strength or area for improvement 12

After Action Report (AAR)/Improvement Plan (IP) • Data from the EEGs and debriefing will be compiled into the AAR/IP. IP 13

Facilitator Evaluator Responsibilities • Be on time • Wear appropriate attire • Make sure you have the tools necessary to perform task • Understand the exercise concept and scenario • Read the sit man 14

Evaluator Responsibilities • Be familiar with the assigned EEGs • Collect information • Make observations on player discussions, knowledge, and intended impact of decisions • Analyze actions and develop post exercise AAR/IP input 15

Pre-exercise Responsibilities • Review appropriate plans, procedures, and protocols • Participate in C/E Briefing • Review appropriate exercise materials to be familiar with scenario, objectives, and core capabilities • Review EEGs and other supporting materials for your area of responsibility 16

Exercise Responsibilities • Monitor and document player performance • Use EEGs to confirm objectives are being met • Focus on critical tasks, as specified in the EEGs and plans being exercised • Do not prompt players or interfere with exercise play • Limited coaching is allowed 17

Evaluator Guidelines At the start of the exercise, evaluators should: • Position themselves to appropriately hear discussions • Be prepared to move between break out groups if necessary to follow various discussions Evaluators should not: • Focus solely on individuals • Interfere with exercise discussion or action 18

Observing the Exercise Evaluators must keep an accurate written record of important issues that take place during the exercise: § Were participants knowledgeable of plans and procedures to meet exercise objectives? § Did discussions validate that operations would occur in accordance with plans and procedures? § Did participants validate that sufficient personnel are equipped, trained, and assigned to perform tasks? 19

Recording Important Events • Evaluators should record events such as: –Discussed actions of players –Key decisions –Discussions that denote deviations from plans and procedures –Player discussion of key movements and activities –Module responses on easel pad –Hotwash responses 20

Postexercise Analysis Responsibilities • Participate in the Hot Wash, and take notes on findings identified by players • Summarize your notes and prepare for the C/E Debriefing • Have your summary ready for the lead evaluator, including bulleted strengths and areas for improvement from activity analysis • Use data to determine whether the exercise capabilities and plans were effectively implemented – Identify observations as strengths or areas for improvement 21

Player Hot Wash • Allows players to participate in a selfassessment of the exercise play • Allows evaluators to clarify additional points and collect missing information • Identify three sustains and three improves • Make sure sustains and improves are listed on the easel pad 22

Controller and Evaluator (C/E) Debriefing • Work to complete EEG analysis forms for submission • Discuss degree to which each objective was completed • Provide strengths, best practices, lessons learned, and possible IP recommendations • Begin outlining issues to be included in AAR/IP on AAR data input forms 23

AAR/IP Input • • Use the EEG AAR data input form Complete one form for each strength and area for improvement. – Observation. Write a one-sentence statement that identifies the strength or area for improvement as noted in the activity analysis tables. – References. Reference relevant plans, policies, procedures, laws, or regulations that could be used to facilitate implementing recommendations for improvement or capturing strengths. – Analysis. Document detailed data on what you observed during the exercise specific to this individual observation. – Recommendations. Suggest steps to allow the participants to further improve the plans or share strengths. • All EEG input is due to Joe or Lou no later than departure from the exercise site on the exercise day. 24

Evaluation Matrix Objective Met Objective Not Met Players followed plan procedures Plan works Capture why plan didn’t work Document best practices and strengths Document how to improve relevant plan elements Players did not follow plan procedures Capture how objective was met Capture why plan was not used Recommend successful actions and procedures be included in plan rewrite May require plan training and familiarization 25

Evaluator Deliverables • Before you leave – Completed EEGs – Notes – Pictures of easel pads in your section (Joe has camera and will do this but easel sheets are important to retain) 26

Questions/Discussion 27