A Recipe With Many Ingredients Design and Implementation

A Recipe With Many Ingredients: Design and Implementation of Culturally Responsive Monitoring and Evaluation System for a Multisite Training Program Gloria Santa Anna, Sonia Lindop, Chris Spicer, Ricardo Gómez

Mass. TERI • Massachusetts Training, Evaluation, and Research Initiative • Mass. TERI contributes to the advancement of women and men in the school nutrition industry by providing innovative, relevant, and costeffective training and professional development solutions.

Mass. TERI Component Short term outcomes Intermediate/long-term outcomes Research Diagnostic of current situation of SNP Increased awareness among policy makers Professional Development Increased skills and competencies of SNP A better prepared workforce that supports the nutritional needs of students Dissemination Increased visibility of program Increased awareness of professional development for SNP Evaluation Improved effectiveness of curriculum, training, and materials Worth and value of the program

Locations

Professional Development • Leadership: supervisory skills, language and cultural sensitivity, selfawareness, team and group dynamics, problem solving and conflict management. • English as a Second Language: Reading, writing, listening and communication in the workplace for employees of all language backgrounds. • Computer Skills: We teach and assess computer concepts and skills so that people can use computer technology in the workplace and in everyday life. Our courses include Word, Excel, e-mail and internet.

Participatory evaluation: the role of the Site Committee • Members of the school nutrition community • Discuss findings • Plan training and evaluation activities • Share and disseminate results

Kirkpatrick’s evaluation model Longer term; Has the training had an impact on the organization/business? Outcome evaluation Are the acquired skills being used in the everyday environment of the learner? Have participants advanced in skills, knowledge, or attitude? Evaluation of training What do they think about the training?

Evaluation of training model Pre-training Needs assessment Q method Selfassessment Start Baseline Intake survey Pre-test Middle Formative Survey Group interviews Finish Summative End-of-course test Survey Retrospective pretest

Retrospective Pre-test Post-test Pretest How good are you at listening to the needs of your staff? 0 1 2 3 4 5 Training Retrospective pretest How good were you at listening to the needs of your staff? 0 1 2 3 4 5 Post test How good are you at listening to the needs of your staff? 0 1 2 3 4 5

Q methodology

Levels of evaluation and tools 1: Reaction 2: Learning 3: Behavior 4: Results Survey X X Questionnaires X X Interviews/Focus groups X X Tests X Case studies X Performance reviews X X Skills/behavior observation X X Action planning X X Action Learning X Business metrics Retrospective pre-post questionnaire X X X

What have we learned? Myths busted. . . • “ The majority of these workers have second jobs. ” • 25% reported having a second job • “Only a small group of ‘motivated employees’ are interested in training…most SNP view training as a waste of time and attended only when mandatory. ” • 73% of the participants, including both PT and FT employees, wanted to receive training • “Older employees (>50 yo) are more resistant to change and not interested in training. ” • People of all ages interested in receiving training. No statistical difference between age groups.

Robust tools for lean evaluation budgets • Lime. Survey (www. limesurvey. org) • Q methodology (http: //qmethod. org/about) • R (http: //www. r-project. org/) • gnuplot (http: //www. gnuplot. info/) • Scribus (http: //www. scribus. net/canvas/Scribus) • La. Tex (http: //www. latex-project. org/) • dot. Project (http: //www. dotproject. net/)

Bar chart with error bars in R (ggplot 2)

R Graphs with ggplot 2

R graphs (cont. )

Dot plot with impulses in gnuplot

Dot plot with confidence intervals in gnuplot

Sample gnuplot code • • • • #x y xlow xhigh location #56 1 49 60 Site-1 #51 2 47 55 Site-2 #50 3 46 56 Site-3 #34 4 30 38 Site-4 #21 5 18 25 Site-5 set yrange [6: 0] set xrange [1: 100] set key off set grid set ylabel "Location" set xlabel "Math score (mean with 95% CI)" set title "Math scores at five locations“ set arrow 1 from 52, 0 to 52, 6 nohead plot "errors 2. txt" u 1: 2: 3: 4: yticlabels(5) w xerrorbars lt 7

To know more… Gloria Santa Anna, Project Director gasantaanna@admin. umass. edu Thank You!