Evaluation Assessment at Duke Engage What Weve Learned

Evaluation & Assessment at Duke. Engage: What We’ve Learned in 10 Years Jacki Purtell One. Duke 2016

Evaluation & assessment is informed by our program model An immersive civic engagement experience – Community-driven – Fully funded – Minimum of 8 consecutive weeks of full-time service – One opportunity per undergraduate Two tracks – Group programs: international or domestic – Independent projects: student designed, international or domestic Multiple learning pedagogies – Structured reflection – Cultural immersion Group Programs: • 40 offered in 2016 with ~385 student spots • Four types: discipline-based, issuebased, community-based participatory research, community-based service placement • Led by Duke faculty or staff or a VSO Independent Projects: • 50 spots available annually • Student designed • Groups of up to 3 students • Requires a faculty or staff mentor • Capstone project upon completion

As well as our mission and values Values • Mission Duke. Engage empowers students to address critical human needs through immersive service, in the process transforming students, advancing the University’s educational mission, and providing meaningful assistance to communities in the U. S. and abroad. • • Duke. Engage values education through immersive civic engagement as the fundamental goal of our work. Duke. Engage values community driven service with our domestic and international community partners as a means to addressing pressing human needs. Duke. Engage values students who pursue their service with humility, respect and curiosity and who seek to understand how global inequalities shape our world. Duke. Engage values ongoing service by its students to local, national and international communities. Duke. Engage values responsible stewardship of financial, environmental and human resources.

How we began Prior to summer 2008 (the first “official” summer): • Assembled a committee of faculty, staff and university stakeholders to create a framework • Used an external consulting firm, The Renselearville Institute, to formulate initial success metrics. • Used the initial success metrics to guide development of profile and instruments 1. 2. 3. 4. 5. 6. 7. 8. 25% of Duke undergraduate students will participate in a Duke. Engage project. 80% of participating students will experience a clear impact from their participation. Of that 80%, one-quarter will experience transformation in their viewpoint or will make different choices in any one of a variety of areas: academic, professional, personal. 95% of Duke. Engage projects are fully implemented or completed. At least 80% of projects produce tangible value for the community. The students applying to and attending Duke University are more diverse and more interested in civic engagement. Duke. Engage is one of the top characteristics used to describe Duke by faculty, staff, applicants and high school guidance counselors. Duke. Engage positively impacts the culture of Duke University with regard to student interaction, intellectual engagement, civic engagement and public service.

Examples of early measures. 25% of Duke undergraduate students will participate in a Duke. Engage project. 80% of participating students will experience a clear impact from their participation. At least 80% of projects produce tangible value for the community. What we mean: In each graduating class, 25% of all students will have participated in Duke. Engage – either as rising freshmen, sophomores or juniors What we mean: Following Duke. Engage, students will articulate a goal change derived from their immersion and service What we mean: Community partners will receive direct or indirect benefits from the service work students due How it maps to mission/value: Duke. Engage values education through immersive civic engagement as the fundamental goal of our work. How we currently measure: Transcript designations granted to participants Duke. Engage empowers students to address critical human needs through immersive service, in the process transforming students. How it maps to mission/value: Duke. Engage values community driven service with our domestic and international community partners as a means to addressing pressing human needs. How we currently measure: Student post-program and sixmonth follow-up surveys – section 1 B (Impacts on you) How we currently measure: The Community Partner Impact Survey - The Student’s Project and Contribution section How it maps to mission/value:

Developed two pathways for evaluation and assessment 1. Administrative Evaluation • One-year cycle • Measures outcomes related to: student enrollment, program and financial management, fundraising, human resources o Anything NOT related to the student or community partner experience • Track according to procedures and metrics outlined by the University Committee for Assessment of Education & Administration Support (CAES) Example • Administrative value: Responsible stewardship of financial resources • Goal: Regularly review financial practices; have a clear policy for enrichment activities, group meals and supplies. • Measure: Program expense logs, pcard forms, yearly budget comparisons, etc.

Example: We spent several cycles seeing that we fell short of our overall enrollment goal for programs • Because of this, we felt we were not always fulling our value of responsible stewardship of human resources. Why? o Annually, receive close to 900 applications for roughly 430 spots o Turn away about 50% of each applicant cohort o When students left slots open, another student could have filled that spot • What we did: o Review historical data § Looked at patterns of attrition for certain (types of) programs § Looked at processes, particularly dates for airline ticketing and visa applications o With a better understanding of student behaviors § Calculate program melt rates § Adjusted acceptance targets up for some programs • Result: Duke. Engage enrollment numbers are more consistent with annual targets

2. Programmatic Evaluation • 18 -month cycle • Measures outcomes related to: student learning in key areas (academic, personal, professional), community partner impact, impact of Duke. Engage on research and teaching. • Evaluate differences: o type of service experience o participant demographics o program elements Model • Three stakeholder groups: o Students o Community partners o Program leadership & staff • Up to three points of contact: pre -service, post-program, 6 -months out • Use paper and Qualtrics platform surveys • Draw on other data sources: • Institutional surveys • Program reports (informal) • Student blogs (informal)

Example: Duke. Engage did not originally view itself as a leadership development program • Our focus was on the cooperative process of immersive service – the equal benefit model between students and community partners. o However, our own mission statement - Duke. Engage empowers students to address critical human needs…transforming students, advancing the University’s educational mission o Students were writing about their leadership development o External stakeholders and funders were asking about leadership outcomes • What we did: o Reviewed and revised early instrument questions o Added and built out questions on leadership development, teamwork, and personal growth in areas such as confidence, outspokenness, etc. o Began to embrace the narrative students were giving • Result: Our story of student transformation relies significantly on gains students attribute to their leadership and teamwork skills • 40% of students report Duke. Engage improves their confidence while 30% report that Duke. Engage helps them be more outspoken • See significant differences in favor of US-based programs and projects

Over the last six years, developed a narrative around student learning and development outcomes • Situate Duke. Engage outcomes within the high-impact practices outlined by George Kuh and others • Establish broad categories for result gathering: o Personal growth o World perspective o Leadership and teamwork skills o Goals reassessment • Establish sub-narratives for different types of service experiences • Ex. Our US-based programs have stronger professional development outcomes, which is consistent with their model of pairing students with community partners, generally in one-onone nonprofit settings.

Integrate aspects of our strategic plan We introduced a six-goal strategic plan in 2012, Duke. Engage 2017: A Blueprint for Deeper and Broader Engagement Goal 1: Curricular Connections Goal 2: Faculty Engagement Goal 3: Community Partner Engagement Goal 4: Expanding Civic Engagement Opportunities in the United States Goal 5: Enrollment Growth Goal 6: Resource Development By 2017, Duke. Engage will provide deeper and broader engagement for all of stakeholders and underscore its value to Duke University. . . http: //dukeengage. duke. edu/about-dukeengage/strategic-plan-2017

Goal 1 Example Duke. Engage will develop prerequisites and follow-up to each experience as part of establishing curricular pathways. What we learned: From a data review, saw that students’ curricular pathways are independent and drive by multiple factors: • Majors/minors/certificates • Estimated free time • Co-curricular commitments How it maps to mission/value: Duke. Engage values education through immersive civic engagement and Duke. Engage values students who seek to understand how global inequalities shape our world. How we currently measure: Student post-program and six-month follow-up surveys – section 2 (Changes you will make/did make) • Collaborative projects with peer departments such a GEO, DEA

Take on special assessments as needed to support plan goals Questions to ask to determine if a special assessment is necessary: • Can we answer this question with our data sources? • Can someone else? o Institutional surveys o Course evaluations • Our goal is to never duplicate ongoing or existing data collection or to contribute to a culture of over-surveying If no: • What is the best way to answer this question? • Are there partners who would benefit from this project? • How can we collaborate with others to maximize the value of the work? Examples of Past Projects: • Duke. Engage 2017 – research to support the strategic plan • Duke. Engage Alumni Survey • Duke Civic Engagement Inventory (with the DOCE) Methods We’ve Used • Surveys • Focus groups • Focused interviews

What we’ve learned through regular assessment 1. 2. 3. 4. 5. 6. 7. 8. 9. Duke. Engage creates social networks on campus More than 75% of students incorporate their experience into writing – academic, blogging, professional 40% of students found their experience made their more confident 85% of students gain collaborative skills 75% of students report their experience influenced their career planning Students refine their academic plans – they are more certain about the issues they would like to explore and their academic pathway 90% of supervisors would partner with Duke. Engage again. 70% of projects are fully finished Students, on average, increase their volunteer or engagement activities by 1 hour per week Students' Reasons for Impact - Overall (Post-Program) Other 21% Gained new perspective 20% Was not life-changing 4% Learned about one self 12% Gained knowledge 5% Was challenging 5% Gained cultural competence Had 5% a new experience 5% Grew personally 8% Gained career perspective/exporsure 7% Gained new friends/relationships 8%

What we’ve learned from special projects: Duke. Engage alumni 1. 2. 3. 4. 5. Alumni remember Duke. Engage as high impact education Duke. Engage significantly shaped undergraduate education with new or different courses, and mindful and purposeful course selection Duke. Engage had a positive impact on alumni career paths – with more alums entering education, health and innovation careers than the average Duke. Engage shaped alum’s perspectives on contemporary global issues Duke. Engage has carried over to influence alum’s volunteer and philanthropic choices My Duke. Engage program provided firsthand experiences with social enterprise work, field-based engineering, and sustainable development. I had just finished my first year at Duke and participated in the Engineering Frontiers FOCUS program, which had classes explicitly named after a few of these topics, but being on the ground actually carrying out the work made this learning much, much more engaging and meaningful. Over the next three years I participated in follow-up classes, at least one of which was sponsored by Duke. Engage, and sought to deepen my experiences. I returned to Uganda after my junior year as an NAE Grand Challenges Scholar and applied many of the basic civic engagement principles I had learned at Duke. Engage. After I turned in my senior thesis, I remember that at first it all felt serendipitous: how my experiences all fit together. But I realize now that Duke. Engage touched off a series of connections and experiences to build on. It was a key life experience. -- E. T. , Uganda 2008

Lessons learned from our work 1. It’s important to establish your evaluation cycle early • It takes time to: gather data, develop a results narrative, determine when and to whom you should report • Sets baselines for your results • Documents program history and initial goals 2. And to share your early results • Forces adherence to a data cycle • Generate feedback and critique of your narrative • Can point to weakness and strengths in your model 3. Both of these contribute to building a culture of evaluation and assessment – which takes time to develop. • Requires reflection on why evaluation & assessment are important and their benefits • Takes time to balance investment in the process with the detachment to see results for what they are.

4. 5. The desire to incorporate all stakeholders doesn’t require that all stakeholders use the same process • Ex. We survey students three times, but community partners only once. This gives us the data we need without burdening our external partners in the process. • It’s not all surveys – we also use focus groups, email and inperson interviews, etc. You don’t know what you don’t know • Important to leave from for the data to speak to you – to let a narrative develop organically • Be alert to new potential data sources and opportunities to gather feedback 6. Evaluation & assessment is an opportunity for self-reflection • For our organization, we learned how others see us and how that aligns with our vision for our work • It’s also an opportunity for reflection for participants – we are trying to do more to share back results, individually and in consolidated reports

Questions and more information For more on the evaluation & assessment we do: http: //dukeengage. duke. edu/about-dukeengage/impact -assessment Contact: Jacki Purtell jacki. purtell@duke. edu 919 -668 -3618