Identification of Child Maltreatment Public Child Welfare Worker

Identification of Child Maltreatment: Public Child Welfare Worker Training Evaluation Outcomes Chris Lee, M. S. W. Maria Hernandez, M. S. W. California Social Work Education Center (Cal. SWEC) University of California, Berkeley (UC Berkeley) S

Acknowledgments Cal. SWEC – UC Berkeley California Regional Training Academies / -University Consortium Inter Training Evaluation Team at Cal. SWEC S S S Cindy Parry, Ph. D. Jane Berdie, M. S. W. Chris Mathias, M. S. W. Barrett Johnson, LCSW Leslie Zeitler, LCSW Salonje Rochell, B. A. S 2

We Will Discuss… S Brief Review of Training Evaluation Literature S Overview and History of Cal. SWEC Training Model S Cal. SWEC’s Use of Embedded Evaluations S Evaluation Analysis Results S Future Directions 3

Training Evaluation Literature S “Training evaluation” refers to a systematic method for assessing or evaluating training S Donald Kirkpatrick’s (1956) levels of evaluation are considered the foundation for many training models used today S The first to establish levels of evaluation for training S Proposed that in order to effectively evaluate training for its impact, it must be sequentially evaluated at 4 levels: S Reaction S Learning S Behavior S Results 4

Training Evaluation Literature S Embedded training evaluation is one approach to evaluating job-related skills (knowledge/behavior) while in the training classroom S Embedded training evaluation is designed to blend instruction with evaluation S Little evidence is available in the literature regarding use and viability of embedded training evaluation in public child welfare training 5

California’s Statewide Training Evaluation Framework S California needed a unified training evaluation system S In response, a statewide collaboration was formed and developed a multi-level evaluation framework S The foundation of this framework is the Common Core training currently required for all new public child welfare workers, which also emerged from this collaboration S California’s current framework is influenced by other training evaluation models 6

Influence of Other Models The American Humane Association Levels of Training Evaluation Parry, C. & Berdie, J. (1999). Training Evaluation in the Human Services. Washington DC: American Public Human Services Association. 7

California’s Statewide Training Evaluation Framework Tracking Attendance Course Evaluation Satisfaction/Opinion Knowledge Skills Transfer of Learning Agency/ Client Outcomes Parry, C. & Berdie, J. (2004). Training evaluation framework report. Berkeley, CA: California Social Work Education Center. Parry, C. & Johnson, B. (2005). Training evaluation in a large system: California Framework for child welfare training evaluation. Proceedings of the Eighth Annual National Human Services Training Evaluation Symposium, 15 -32. 8

Training: Identifying Child Maltreatment S There are 7 total core training modules for public child welfare workers in California S Depending on the region, some workers attend this training during pre-service (before beginning work in the field), while others attend in-service (shortly after starting work in the field) S Training module: Child Maltreatment Identification (I, II) S Identifying neglect, physical abuse, emotional abuse (CMI-I) S Identifying sexual abuse & exploitation (CMI-II) S Each CMI module is 1. 5 days long, involving instruction and an embedded evaluation administered at the end 9

Embedded Evaluations In The Classroom S Embedded evaluations (one for each CMI module) consist of 4 scenarios that may or may not describe a case of maltreatment S CMI-I evaluation assesses ability to identify physical abuse S CMI-II evaluation assesses ability to identify sexual abuse S Trainees read case scenarios on their own and answer a series of questions for each scenario, ultimately deciding in each case if maltreatment did or did not occur S Embedded evaluations are administered on non-carbon copy (NCR) paper 10

Embedded Evaluations In The Classroom S Trainees are given an allotted amount of time to complete the embedded evaluation (EE) S Instructor collects completed EEs, then reviews case scenarios and test questions with trainees and engages them in discussion S EEs are generally used for evaluating skill; however the EEs used in the CMI modules evaluate both knowledge and skill 11

Data and Methods S Current analysis: Explored associations between trainee characteristics (California Common Core Curricula Demographic Survey) and CMI embedded evaluation results S Utilized logistic regression to analyze trainee characteristics and their likelihood of passing CMI embedded evaluations S “Passing” was defined as correctly identifying whether maltreatment did or did not occur on at least 3 out of 4 case scenarios S Time frame of data collection: January 1, 2007 to December 31, 2008 S Final Sample Sizes S CMI-I (n = 1282) S CMI-II (n = 896) 12

Demographic Handouts 13

CMI-I and CMI-II Evaluation Results 14

Embedded Training Evaluation and “Chain of Evidence” S Review: As indicated in the literature, EEs are useful for instruction, evaluation and skill building S Cal. SWEC: EE results are used to inform revisions to curricula, training and future knowledge/skill evaluations S Use of EEs help to strengthen the skills level of the training evaluation framework, moving closer to the next level: transfer of learning 15

Future Directions: Next Links In The Chain S Further explorations of existing data in order to strengthen foundational training components such as: curriculum, training in the classroom S Continue to expand the efficacy of each level (e. g. , revise the CMI-I (EE) to also assess neglect) S Integrate training in collaborative research projects (e. g. , CMI-II training & Child Forensic Attitude Scale) S Work through remaining levels of the training evaluation framework and eventually reach the final link in the chain: influencing client/agency outcomes 16

References Cal. SWEC. (In Press). Evaluation of the California common core for child welfare training, implementation status, results and future directions [White paper]. Berkeley, CA: California Social Work Education Center. Kirkpatrick, D. (1959). Techniques for evaluating training programs. Journal of the American Society of Training Directors, 13(3 -9), 21 -26. Mc. Cowan, R. J. & Mc. Cowan, S. C. (1999). Embedded evaluation: Blending training and assessment. Buffalo, NY: Research Foundation of SUNY/Center for Development of Human Services. Retrieved from http: //www. eric. ed. gov/ERICWeb. Portal/custom/portlets/record. Details/detailmini. jsp? _nfpb=true&_&ERICE xt. Search_Search. Value_0=ED 501715&ERICExt. Search_Search. Type_0=no&accno=ED 501715 Parry, C. & Berdie, J. (1999). Training Evaluation in the Human Services. Washington DC: American Public Human Services Association. Parry, C. & Berdie, J. (2004). Training evaluation framework report. Berkeley, CA: California Social Work Education Center. Parry, C. & Johnson, B. (2005). Training evaluation in a large system: California Framework for child welfare training evaluation. Proceedings of the Eighth Annual National Human Services Training Evaluation Symposium, 15 -32. 17

For more information on training evaluation in California please visit the Cal. SWEC website: http: //calswec. berkeley. edu/Cal. SWEC/CWTraining. html Leslie Zeitler, Training and Evaluation Specialist lzeitler@berkeley. edu Maria Hernandez, Graduate Student Researcher hernandezm@berkeley. edu Chris Lee, Graduate Student Researcher clee 07@berkeley. edu S 18