OUTCOMES OF A CHARACTER EDUCATION PROGRAM IMPLICATIONS FOR

OUTCOMES OF A CHARACTER EDUCATION PROGRAM: IMPLICATIONS FOR SCHOOL DISTRICTS WITH INCREASINGLY LIMITED RESOURCES Presented By: Amy Schelling, Tammy De. Roo and Dr. Gary Miron

SETTING THE EVALUATION STAGE… Research supports that positive character development increases student achievement and decreases behavioral issues in the school setting. Much of the research related to the efficacy of character education programs have focused on implementation in the school setting, no studies include character education programs delivered off-site in a residential setting. Given the current educational context of dwindling financial resources and increased accountability through high-stakes testing, school district officials must justify using resources to support character education as well as lost academic instructional time.

A BRIEF BACKGROUND OF THE PROGRAM EVALUATED… Character Education Program: Off-site Experiential-based Residential (2, 3, or 4 -day experience) Led by trained staff external to the school Content: HCRR & Safety

THE PURPOSE OF THE PILOT STUDY… The Outdoor Center partners with regional schools to provide a character education program to a diverse population of students. Through experiential-based activities, students learn to incorporate the principles of Honesty, Caring, Respect, and Responsibility (HCCR) into their daily lives. The purpose of the evaluation was to specifically conduct an evaluation of the Character Education component of the program that focused on identifying whether the program was meeting its stated program goals and anticipated outcomes for students and teachers.

THE CONTEXT WITHIN WHICH THE EVALUATION WAS SITUATED… Requested by camp staff Executive Director’s primary objective was to justify requests for additional/continued funding The evaluation was both formative and summative, however. Why not just summative? Program had never before been evaluated for its processes. Evaluators provided education to camp staff regarding the likelihood of improved utilization of evaluation results containing both formative and summative conclusions.

WHAT WE LEARNED…IT CAN BE DONE! Methodology: Quasi-experimental two experimental schools who partici-pated in the program and one control school that does not Theory-driven approach Outcomes Logic Model Conceptualizing the programs primary components Framework for identifying anticipated outcomes of the program Qualitative & Quantitative methods of data collection School Climate Scale, Post-camp Surveys, teacher and program staff interviews, self-reported data from diverse stakeholders Use of existing program data Cost to provide, scholarships, school cost per student, fundraising capacity Number of staff, participating schools, participating students Both standardized & non-standardized instruments School Climate Scale, interview protocols, post-camp surveys

METHODOLOGY… We incorporated the use of both quantitative and qualitative methods of data collection. The evaluation was largely dependent on existing program data and self-reported outcomes from diverse stakeholders However employing a quasi-experimental design allowed us to see the nature and relative size of changes in school climate before and after two schools participate in the character education program. These changes were then compared with a control school that did not participate in the program.

GRAPHICAL REPRESENTATION OF STUDY Experimental Schools Control School PRETEST INTERVENTION POSTEST Student & Teacher Surveys Camp Experience Student & Teacher Surveys

WHAT CAN BE OBTAINED FROM AN EVALUATION SUCH AS THIS? Given the quasi-experimental design and the context and timeframe in which the evaluation took place, stakeholders can expect… Formative aspects Implementation Is the program being implemented as designed? Is the program meeting goals related to program outputs? What is the level of quality in service delivery of the program? Summative aspects Are students meeting the intended program outcomes (immediate/intermediate)? Are teachers meeting the intended program outcomes (immediate/intermediate)?

WHAT CANNOT BE OBTAINED FROM AN EVALUATION SUCH AS THIS? Summative: Evaluating if intermediate and long term student outcomes have been met Evaluating if intermediate and long term teacher outcomes have been met Evaluating if a Causal link between program implementation and intermediate and long term student and teacher outcomes exists (behaviors/internalization) Evaluating if a link exists between changes in school climate or student self-esteem (due to relatively short duration of study and other character education programs offered simultaneously in schools) due to program A comparison to other character education programs

COSTS AND RELATED CONSIDERATIONS When contracting for an evaluation, organizations dictate the scope and depth of the evaluation depending on the resources they devote for the evaluation. The organization that requested this evaluation petitioned university administrators to conduct this evaluation as a component of its community service. In the end, we took on the responsibility to conduct the evaluation as a part of a graduate Evaluation Practicum course. The organization that requested the evaluation did provide some support to the team by copying instruments. Had this evaluation been undertaken by an independent consultant, it probably would have costs about $12 -$15 K.

IMPLICATIONS OF OUR STUDY… With increased pressure on schools in terms of accountability standards, the use of limited time and funding becomes a potential issue in the decision of school administrators to devote resources and academic time to provide students with the opportunity to participate in an off-site character education program. The results of this evaluation suggest that participation in the off-site, residential camp character education program was beneficial to participants in several ways and is likely an appropriate use of school resources.

Unintended Outcomes… Throughout the evaluation process, we continually sought to learn if the program was resulting in any positive or negative outcomes that were unintended. The evaluation did not discover any specific instances in which camp participation delayed, or otherwise diverted, limited resources in a manner that reflected poorly on student achievement. These results may help to inform school administrators in future decision-making.

UNINTENDED OUTCOMES (CONT. ) Another potential unintended outcome that we examined throughout the evaluation process was the potential for the camps character education curriculum to conflict with the existing school-based character education programs, creating confusion or disruption for students. Our findings did not reflect any significant contradiction between the camp’s character education programs and those that are being taught in the schools. As program staff were interviewed, it became clear that these individuals have a high sense of ownership over the program and its mission. It appears that these individuals have truly internalized the HCRR principles into their own lives and interactions with others. In that many of the program staff students majoring in Education, they have the potential to share this positive approach with countless future students

THINGS TO CONSIDER… Based on our experience, when conducting such an evaluation consider… Logistics of data collection Pilot instruments or use standardized instruments Understand how stakeholder intends to use information and design study in manner that will produce useable results Educating stakeholder about realistic expectations, etc. Better understand teacher obligation(s) to program – i. e. , we didn’t know some teacher s didn’t stay Seek ways to identify/eliminate sources of bias

SUGGESTIONS FOR FURTHER RESEARCH? AVENUES OF STUDY Compare two or more programs Longer study (extend data collection period) Longitudinal? Larger sample/variety of survey participants (i. e. purposive sampling of teachers – previous years of experience, etc. ) Triangulate data sources (parents & teachers that didn’t attend) See report

REFERENCES Best Practice Briefs. (2004). School climate and learning. No. 31. University Outreach & Engagement, Board of Trustees of Michigan State University. Brown, K. M. , Anfara, V. A. , Jr. , & Roney, K. (2004, August). Student achievement in high performing, suburban middle schools and low performing, urban middle schools: Plausible explanations for the differences. Education and Urban Society, 36(4), 428 -456. Cohen, J. , Mc. Cabe, E. M. , Michelli, N. M. , & Pickeral, T. (2009). School climate: Research, policy, practice, and teacher education. Teachers college record, 111(1), pp. 180– 213. Curtis-Seinik, C. , Mc. Carthy, M. , Nadal, K. , Pfeiffer, D. , Tella, A. , & Wagner, D. (2006). Character education, K-12, in Uniondale. Social Studies and the Young Learner, 19(1), 27 -29. Kuperminc, G. P. , Leadbeater, B. J. , & Blatt, S. J. (2001). School social climate and individual differences in vulnerability to psychopathology among middle school students. Journal of School Psychology, 39(2), 141– 159. Kuperminc, G. P. , Leadbeater, B. J. , Emmons, C. , & Blatt, S. J. (1997). Perceived school climate and difficulties in the social adjustment of middle school students. Applied Developmental Science, 1, 76– 88. Rubin, R. (2004). Building a comprehensive discipline system and strengthening school climate. Reclaiming Children and Youth, 13(3), 162 -169. Skaggs, G. , & Bodenhorn, N. (2006). Relationships between implementing character education, student and student achievement. Journal of Advanced Academics, 18(1), 82 -114. Smith, M. R. (2006). Contemporary character education. Principal Leadership (Middle School Ed. ), 6(5), 16 -20. behavior,