The Pell Institute For the Study of Opportunity

The Pell Institute For the Study of Opportunity in Education Building Evaluation Capacity in College Access Programs AEA Evaluation 2011: Values and Valuing in Evaluation The Hilton Anaheim 777 Convention Way, Anaheim, CA Thursday, Nov 3, 2011 11: 40 AM to 12: 25 PM La Jolla Chandra Taylor Smith, Ph. D. Vice President for Research, COE Director, The Pell Institute 1

The Pell Institute For the Study of Opportunity in Education Building Evaluation Capacity in College Access Programs The Need for Evidence 2

The Pell Institute For the Study of Opportunity in Education Evidence For over half a century, a succession of Congresses and presidents has sought to do two things for American elementary and secondary education: raise overall achievement, and narrow the gaps between high- and low-income students as well as between minority and white students. The federal government has spent roughly $2 trillion on these efforts since 1965, adjusting for inflation. -Andrew J. Coulson, Director, Center for Educational Freedom at the Cato Institute, The Impact of Federal Involvement in America’s Classrooms, Feb. 10, 2011 3

The Pell Institute For the Study of Opportunity in Education Evidence “We have little to show for the $2 trillion in federal education spending of the past half century. ” -Andrew J. Coulson 4

The Pell Institute For the Study of Opportunity in Education The Evaluation Toolkit Ø Collaborative venture between the Pell Institute and Pathways to College Network Ø Received funding from the Bill and Melinda Gates Foundation and the Lumina Foundation for Education 5

The Pell Institute For the Study of Opportunity in Education The Evaluation Toolkit Ø It is an online, step-by-step guide to program evaluation specifically designed to help college access and success professionals to: 1. Determine the effectiveness of program practices, and 2. Generate recommendations for program improvement and success. Ø It is freely accessible on the Pell Institute Website at: www. pellinstitute. org, click on Resources tab 6

The Pell Institute For the Study of Opportunity in Education Program Level Evaluations Ø Build the capacity for program level evaluation Ø Program Level evaluation models § Empowerment § Culturally Responsive § Collaborative § Participatory § Development 7

The Pell Institute For the Study of Opportunity in Education Empowerment Evaluations Ø Engage evaluation concepts, techniques, and findings to foster improvement and self-determination Ø Employ both qualitative and quantitative methodologies Ø Focus usually on programs, though can also be applied to individuals, organizations, communities and societies or cultures 8

The Pell Institute For the Study of Opportunity in Education Culturally Responsive Evaluations (CRE) ØDemographics, sociopolitical perspectives, contextual dimensions, and characteristics of culture are given primary consideration ØThe lived experience especially from communities and populations of color is viewed to be significant Ø“Evaluating down” is avoided and the engagement of stakeholders (e. g. the voice of students, etc. ) to assist in identifying the purpose, framing the questions, analyzing data, disseminating evaluation results, etc. is valued 9

The Pell Institute For the Study of Opportunity in Education Advantages of Program Level Evaluations Development of an Evaluation Mindset and Development of an Evaluation Culture 10

The Pell Institute For the Study of Opportunity in Education An Evaluation Mindset Shift in Thinking Assessment Thinking Evaluation Thinking Only collecting & drawing meaning from descriptive data about program Now seeking & drawing meaning about the quality of the program 11

The Pell Institute Achieving an Evaluation Culture The development of an evaluation mindset For the Study of Opportunity in Education The development of an evaluation practice which strives to achieve program quality in a culture of excellence 12

The Pell Institute For the Study of Opportunity in Education An Evaluation Culture An evaluation culture strives to achieve program quality and excellence by regularly implementing Formative & Summative Evaluation Practices 13

The Pell Institute For the Study of Opportunity in Education An Evaluation Culture Questions Formative Ø How are we doingon-going? Ø How can we do things better? Ø Are our resources and activities adequate to achieve our goals? Ø How will we implement changes that need to happen now? Summative Ø Were our program objectives met? Ø Will we need to improve and modify the overall structure of our program? Ø What is the overall impact of our program? Ø What resources will we need to address program’s weaknesses? 14

The Pell Institute For the Study of Opportunity in Education Inside the Evaluation Toolkit COMMUNICATE & IMPROVE ANALYZE DATA EVALUATION 101 DEVELOP A PLAN COLLECT DATA 15

The Pell Institute For the Study of Opportunity in Education EVALUATION 101 Ø Evaluation 101 is the introductory section of the Toolkit. Ø This guide provides foundational information about: Ø EVALUATION MYTHS AND BENEFITS Ø EVALUATION APPROACHES AND TYPES Ø TYPES OF DATA Ø RESOURCES AND REFERENCES 16

The Pell Institute For the Study of Opportunity in Education DEVELOP A PLAN Ø In order to ensure “quality” evaluation results, proper planning is required. Ø The content in this guide helps users to: ØSELECT A LEAD EVALUATOR AND TEAM ØENGAGE STAKEHOLDERS ØDEVELOP EVALUATION QUESTIONS ØCREATE A LOGIC MODEL ØCHOOSE AN EVALUATION DESIGN ØDEVELOP A STRATEGY AND BUDGET 17

The Pell Institute For the Study of Opportunity in Education COLLECT DATA Ø Before starting to collect data it is necessary to: Ø Revisit developed evaluation questions Ø Determine what resources are available ØThe content in this guide helps users to: Ø UNDERSTAND THE RANGE OF DATA TYPES, SOURCES, AND COLLECTION METHODS Ø HOW TO IDENTIFY WHAT COLLECTION METHOD(S) ARE BEST Ø DETERMINE WHAT SAMPLE SIZE IS APPROPRIATE 18

The Pell Institute For the Study of Opportunity in Education ANALYZE DATA Ø The toolkit supports a “mixed method” approach to collecting data which produces a more comprehensive evaluation Ø The Analyze Data section provide guidance on: Ø HOW TO CONVERT RAW DATA INTO USABLE INFORMATION Ø WHAT TECHNIQUES CAN BE USED TO ANALYZE QUANTITATIVE DATA Ø WHAT TECHNIQUES CAN BE USED TO ANALYZE QUALITATIVE DATA Ø WHICH RESOURCES AND COMPUTER SOFTWARE PROGRAMS CAN BE USEFUL 19

The Pell Institute For the Study of Opportunity in Education COMMUNICATE & IMPROVE Ø To often, many programs evaluate their efforts and do very little with the findings of the evaluation. Ø This final section of the Toolkit is dedicated to helping users to think about how their evaluation results can be disseminated. ØThe content in this guide helps to: Ø DEVELOP THEIR “COMMUNICATION” PLAN Ø IMPROVE THEIR PRACTICES BASED ON THE EVALUATION FINDINGS 20

The Pell Institute For the Study of Opportunity in Education Personalize an Evaluation Toolkit Webinar Register your team to have a personalized “evaluation encounter” with any of the Evaluation Toolkit guides: ü Evaluation 101: Developing an Evaluation Mindset ü Develop a Plan: How to Create a Logic Model ü Collect Data: Determining the Right Method ü Analyze Data: Quantitative and Qualitative Techniques ü Communicate and Improve 21

The Pell Institute For the Study of Opportunity in Education Personalize an Evaluation Toolkit Webinar Ø Provide The Pell Institute with the actual questions, data, and other materials your team is already working on ten days prior to the scheduled webinar and your individualized evaluation information will be featured as the focus of the webinar. Ø The basic price range per Personalized Evaluation Toolkit Webinar: q $445. 00 per connection for one personalized 90 minute webinar q Multiple 90 minute personalized evaluation toolkit webinars are available at a discounted rate upon request 22

The Pell Institute For the Study of Opportunity in Education Thank You Q&A www. pellinstitute. org Access Evaluation Toolkit Dr. Chandra Taylor Smith chandra. taylorsmith@pellinstitute. org 202 -638 -2887 ext. 207 23