EZIZ Online Training Longitudinal Evaluation Leslie Heyden Natalie

EZIZ Online Training: Longitudinal Evaluation Leslie Heyden, Natalie Preas James Huang, Catherine Salibay, Claudia Aguiluz NIC— March 28, 2011

eziz. org

What we will talk about today. • What is eziz. org? • What lessons are in the EZIZ Training Curriculum? • What is longitudinal evaluation? Why is it important? • How is EZIZ data collected? • What’s new on EZIZ? • How can you benefit from EZIZ?

Longitudinal Evaluation: • Is an approach that continuously collects the same statistical data over time • Is used for identifying patterns and changes, e. g. increases or change in usage • Produces data that the user can “slice and dice” many ways, e. g. § Look at a snapshot/single moment in time § Sort learners by job category, state, weather they work for a California VFC provider If the data have been systematically gathered, it’s easy to go back and do “detective work”, e. g. “When did we see an increase in pharmacists using the training? ”

EZIZ Data Sources and Data Collection and Analysis Tools • Google Analytics: # of visits to eziz. org over time • EZIZ Learning Management System (LMS): learner and lesson data for EZIZ training • SAS (Statistical Analysis Software) • Emails from users (qualitative data)

eziz. org

Materials and Resources

Job aids

EZIZ Milestones • 2008 • eziz. org website launched • First 2 lessons of curriculum—Preparing Vaccines and Administering Vaccines— launched • 2009 • H 1 N 1 pandemic • EZIZ lessons Preparing Vaccines and Administering Vaccines used across the country to train vaccinators • Storing Vaccines lesson launched

EZIZ Milestones (cont’d) • 2010 • Transporting Refrigerated Vaccines job aid developed and posted on eziz. org • Pertussis epidemic • EZIZ website hosts many pertussis-related materials and job aids • Two new lessons and their related job aids launched : • Monitoring Refrigerator Temperatures • Monitoring Freezer Temperatures • 2011 • By March, more than 15, 000 people had registered to use EZIZ training

Weekly Web Stats

EZIZ Training: Background • Limited VFC field staff time to train VFC providers and staff • Limited availability and access to classroom IZ training for medical assistants • Continued concerns about storage and handling

Needs Analysis–Objectives • Identify primary target audience (medical assistants) characteristics • Identify MAs’ IZ training needs • Identify additional VFC provider training/information/education needs • Assess provider technological readiness (i. e. Is distance learning feasible? )

Needs Analysis–Methodology • Send survey with VFC recertification letter (received 1000+ responses) • Interview § § § VFC field reps VFC call center customer service reps VFC program manager VRC field services manager Medical assistants Instructors in medical assistant training programs • Review existing training resources (print, classroom, online)

Needs Analysis–Conclusions • Distance learning is feasible • elearning is optimal training delivery method • Training is needed in these content areas: § § VFC program info Vaccine administration Storage and handling Vaccine inventory management

EZIZ Training

EZIZ is “high-end” elearning • • Animation Full narration Creative graphics Avatars Short and concise Only “must-have info”— “no nice-to-know” Highly interactive so that the learner is fully engaged Is not an “electronic page turner!”

EZIZ: Storing Vaccines Lesson

EZIZ Learning Management System (LMS) • Is a customized database that automates the administration of the EZIZ online training • Is the primary data source for monitoring & evaluating the training • Registers users, tracks the lessons in the curriculum • Records pretest, lesson completion, and posttest data from learners • Is data source for learner and training usage reports

Every State Uses EZIZ Training

Qualitative Data • “Great, highly recommend for medical staff to review. Straight to the point content, very clear. ” (D. M. , Disease Control Nurse Educator, California) • “You have done another great job! We require that new employees complete all lessons during our new employee orientation and during all our community classes and for any students (nursing and medicine) prior to attending immunization clinics. Thank you for developing this teaching/learning system and for making it available to everyone. ” (Large health care provider group, Washington) • “I am using the EZ-IZ training to train LVNs and MAs on vaccine administration and storage and handling. They love it!” (Large HMO, California)

Current California VFC Program Activities • Efforts to increase use of EZIZ training: • Hosting VFC Program’s core activities, such as on-line recertification and flu ordering, to get VFC providers closer to EZIZ • Getting creative about handing vaccine losses or storage and handling incidents • $$$ compensation vs. lesson completion • Completing new temperature monitoring lessons for a free thermometer!

Newest Lessons • Monitoring Refrigerator Temperatures—how to: § Read a MIN/MAX thermometer § Identify unacceptable refrigerator temps § Complete a refrigerator log • Monitoring Freezer Temperatures—how to: § Read negative numbers § Identify unacceptable freezer temps § Complete a freezer log

Job aids: Monitoring Temps

Check it out!

Leslie Heyden Health Educator & Instructional Designer Leslie. Heyden@cdph. ca. gov (510) 620 -3754 Natalie Preas Creative Director & Project Manager for eziz. org Natalie. Nakahara@cdph. ca. gov (510) 620 -3750 www. eziz. org