Evaluability Assessment Understanding the Practicality and Utility of

Evaluability Assessment Understanding the Practicality and Utility of an Evaluation Thomas E. Grayson & John A. Evans University of Illinois at Urbana-Champaign (tgrayson@uiuc. edu) (jaevans 7@illinois. edu) American Evaluation Association Conference, October 2014, Denver, Colorado

Today’s Bottom Line An understanding of evaluability assessment Five minute review – Purposes of Program Evaluation Evaluability Assessment – Definition – Its rationale – Its action areas Call to Action

Five Minute Review: Primary Purposes of Program Evaluation 1. Process Evaluation (e. g. , improvement) 2. Outcome Evaluation (e. g. , accountability) 3. Developmental Evaluation (e. g. , innovation) 4. Learnings - Knowledge Generation (e. g. , barriers & enablers) 5. Forward Thinking / Focused Evaluation (e. g. , sustainability)

Understanding Evaluability Assessment (Action Areas) 2. Visions, Missions & Goals from different altitudes 3. A Program’s Logical Framework 1. Stakeholders & Boundary Partners (Need for Evaluation Information) 4. Valued evidence-based future

What is Evaluability Assessment? (a. k. a. Exploratory Evaluation) It is a tool that actively involves key stakeholders in answering the question of whether a program is ready for meaningful, useful evaluation. Evaluability assessment is about exploration and discovery.

A Decision-oriented Approach An evaluability assessment is a decision-oriented approach to evaluation that provides results for determining the best way to produce worthwhile evaluations. A quality evaluability assessment results in: • a program theory model, which represents the logic of how a program works; • findings about the alignment of program theory and program reality; • related findings about the plausibility of a program-yielding positive results as conceived and implemented; • recommendations for filling in any gaps in alignment and increasing program plausibility; and • recommendations for further evaluation. (Trevisan, M. S. , & Walser, T. M. , (2015). Evaluability Assessment: Improving Evaluation Quality and Use. Sage Publication

Why EA? Three Big Reasons! Evaluability assessment is a starting point, readying you for an evaluation (e. g. , program review, accreditation, outcomes assessment). The reality: Administrators and managers have a need to manage the meaning of programs (why they exist and how they work) and a need to manage program information. Evaluability assessment will enhance the likelihood of using evaluation information, findings and recommendations.

More Reasons! § Stakeholder involvement is an essential element of EA. It identifies stakeholder needs, concerns, and can ensure that diversity of perspectives and values are included in evaluation § EA can surface disagreements among stakeholders about program theory, design and/or structure § EA can determine plausibility of the program design/model § EA can highlight the need for changes in program design § EA promotes understanding of program culture and context § EA supports organizational learning and evaluation capacity building

Overall Rationale for Conducting an EA The overall rationale for conducting an EA is to improve program performance through meaningful and useful evaluation. When appropriate evaluation criteria have not been identified or the intended uses of evaluation are unclear, evaluability assessment can be very useful. Evaluators begin by identifying: § the big picture evaluation question; § program activities and goals; § performance indicators (types of evidence) by which the program will be evaluated; and § the data sources to be used for data collection, comparisons, and analyses. (Wholey, 2004).

Typical Beginning: Big Picture Evaluation Questions Provide Focus for an Evaluability Assessment Process evaluation (Quality of content, design, and delivery or implementation) What was the quality of the program’s content/design and how well was it implemented? Learnings (barriers and enablers, nuances, surprises, causal explanations / mechanisms / feedback loops, plausible explanations) What were the barriers and enablers that made the difference between successful and disappointing implementation and outcomes? Outcome evaluation (value of outcomes) Was the program worth implementing? Did the values of the outcomes outweigh the value of the resources used to obtain them? Forward thinking / focused (evaluation questions (e. g. , exportability, sustainability, threats, opportunities) How strong is the program’s sustainability? Can it survive/grow in the future with limited additional resources? 2013, E. Jane Davidson, Actionable Evaluation Basics

Typical issues discovered during an evaluability assessment § Key stakeholders and intended users of evaluation information fail to agree on the goals, side effects, and performance criteria to be used in evaluating the program § Program goals are found to be unrealistic given the resources that have been committed to them and the program activities that are under way. § Relevant information on program performance is often not available § Administrators on the policy or operating level are unable or unwilling to change the program on the basis of evaluation information

Elements To Be Explored Program Design Evaluability Assessmen t Key stakeholders Context & Utility Data Availability EA is tool that helps managers verify whether these elements are in place ensuring a reliable and credible evaluation. UNIFEM Evaluation Unit, 2009. “Guidance Note”.

Action Area 1: Establish a key stakeholder work group (intended users of evaluation information), and identify and collect documents. Program Stakeholders & Boundary Partners – Terminology used by IDRC

Community College Program Stakeholders & Boundary Partners r ist D o C ict Other CCs ity n u m m Public Schools Other College Faculty College Advancement & Planning College Faculty Mentors CC Program Manager Illinois Board of Higher Education Corporate Partners Program Staff Administrators Academic Affairs Students Administrators Student Services Alumni Illinois CC System Federal Funding/Grants

Action Area 2. Clarify the program design (i. e. , intent) from the perspectives of key stakeholders; identify intended outcomes. A Program’s Logical Framework – It’s Design

Unsound or Weak Program Design/Theory If the program design or theory is not sound, there is little reason to assess and evaluate to answer any evaluation questions regarding the program’s implementation, its outcomes, impact, value or worth, or its cost-effectiveness. Evaluability assessment can reveal deficiencies in the program design and be helpful in making improvements before time and costs are invested in the conduct of an evaluation. 17

Include both “practitioner” logic & “research logic” (When Clarifying Program Design) Practitioner Logic § § § Well-grounded in local knowledge and context More accurate description of a program Facilitates evaluation use Facilitates evaluation capacity building Supports evaluation credibility Research Logic § § § Best practices inform program activities and links between inputs, activities, and outcomes Aids in interpretation and generalizability of evaluation findings Addresses funding requirements for researchbased practices Facilitates program scaleup Supports evaluation credibility

A Program’s Logical Framework Get a. k. a. Outcomes Do Resources We use these resources… Core Components to do these things in the amount of … (Number) (Frequency) (Dosage) (Duration) Clients Expect to See Like to See Love to See for these clients… to change them so they can do in these ways… these things… so ultimately issues are addressed. (Beneficiaries) (Knowledge) (Skills) (Attitudes) (Opinions) (Motivations) (Behavior) Assumptions, Contextual & External Factors (Condition)

Discussion Purposes Only UIC CHANCE Program Our Vision, Values and Mission Vision & Values [name], Here are some guiding questions for developing a vision statement. First, identify the CHANCE Program’s core values? What are they? What is the clear image of what the CHANCE Program could and should become if it were to realize its full potential? This is an image of how things could be – what the Promised Land looks like. Mission [name], Guiding questions for developing a mission statement: • What function does the CHANCE Program serve? • For whom does the CHANCE Program serve this function? • How does the CHANCE Program go about filling this function? • Why does the CHANCE Program exist? That is, what broad social need does it serve? [name], this is a rough draft or your mission statement. The mission of UIC’s CHANCE Program is to seek and recruit strong promising underrepresented undergraduate candidates from Chicago and Suburban communities for admission to UIC, and to provide the necessary academic and developmental supports that will enable them to persist and graduate prepared for the challenges of the 21 st century.

UIC CHANCE Program and Boundary Partners n ba r bu s u S tie & ni o u g ica omm h C C Chicago Public Schools UIC Academic Center for Excellence UIC CHANCE Program Communitybased Partners UIC Colleges City Colleges of Chicago n ba r bu s u S tie & ni o u g ica omm h C C = CHANCE Program`s Boundary Partners 21

UIC CHANCE Program Level I Logic A UIC program for increasing and graduating underrepresented students ü Strong admission applicants ü Incoming freshmen ü Transfer students ü Probationary students Primary Resources Administrative & Professional Staff Students UIC Colleges UIC resources Chicago Public Schools City Colleges of Chicago Community-based Partners Space (e. g. , offices, conference room, tutoring room, and computer stations) Long Term Strategic Goal UIC Chance Programs/Services I. Recruitment and Outreach Summer Institute Program Chicago Public Schools – Recruitment Initiatives City Colleges of Chicago – Financial aid, College transfer and goal setting workshops Community-based Outreach Initiatives II. Academic Support Partnerships Academic Counseling UIC Math Learning Center UIC Writing Learning Center UIC Science Learning Center Financial aid Study Skills Enhancement III. Student Development Programs Health & Wellness Program Ambassador Program – Mentoring & Cultural Enrichment Events High numbers of underrepresented UIC students graduating with a foundation of knowledge, skills, attributes and commitment necessary for leadership and citizenship in an increasingly diverse, technologically sophisticated, and complex global society.

UIC CHANCE Program Level II Logic Primary Resources UIC Chance Program Strategies/Services Strategic Outcomes Expect to See Administrative Recruitment and Outreach • Summer Institute Program • CPS - Recruitment Initiatives • CCC - Financial aid, College transfer and goal setting workshops • Community-based Outreach Initiatives Academic Support Partnerships • Academic Counseling • UIC Math Learning Center • UIC Writing Learning Center • UIC Science Learning Center • Financial aid • Study Skills Enhancement Student Development Programs • Health & Wellness Program • Ambassador Program – Mentoring & Cultural Enrichment • Probation Outreach Increased awareness of the UIC undergraduate experience by students in the CPS and CCC Services Professional Staff Students UIC Colleges UIC Academic Center for Excellence Chicago Public Schools City Colleges of Chicago Community-based Partners Equipment/Technolo gy Space (e. g. , offices, conference room, tutoring room, and computer stations) Increased enrollment of underrepresented students (e. g. , incoming freshmen and transfer students) Students learn new skills (e. g. , technology, study skills) for persisting in their academic program Students experience increased satisfaction and comfort in the UIC community Increased number of students participating in health & wellness programs (e. g. , physical, social, emotional, environmental, etc. ) Students increase their awareness of cultural differences and social justice & equity issues Increased awareness and engagement in leadership Strategic Outcomes Like to See Increased student personal and academic goals attainment – acquisition of an academic identity Increased retention of underrepresented UIC undergraduate students (focusing on 1 st and 2 nd year undergraduate students, new transfer students from community colleges, and undergraduate students on academic probation) Increased persistence and graduation rates of underrepresented undergraduate students Strategic Outcome Love to See High numbers of underrepresented UIC students graduating with a foundation of knowledge, skills, attributes and commitment necessary for employment, leadership and citizenship in an increasingly diverse, technologically sophisticated, and complex global society

UIC CHANCE Program for Underrepresented Students Spheres of Program Influence on Achieving Outcomes P R O G R A M R E S O U R C E S & S T R A T E G I E S Expect To See Outcomes Changes in behavior, relationships, & attitudes of underrepresented students • Increased awareness of the UIC undergraduate experience by students in the CPS and CCC • Increased enrollment of Outputs underrepresented students (e. g. , incoming freshmen and transfer What students) engage in & receive from UIC CHANCE • Students learn new skills (e. g. , technology, study skills) for persisting in Program their academic program Direct control • Students experience increased satisfaction and comfort in the UIC community • Increased number of students participating in health & wellness programs (e. g. , physical, social, emotional, environmental, etc. ) • Students increase their awareness of cultural differences and social justice & equity issues • Increased awareness and engagement in leadership development and community service • Students increase their awareness and knowledge of civic & professional ethics Strong influence Like To See Outcomes Love To See Outcome Changes in status of UIC underrepresented students High numbers of underrepresented UIC students graduating with a foundation of knowledge, skills, attributes and commitment necessary for employment, leadership and citizenship in an increasingly diverse, technologically sophisticated, and complex global society • • • Increased enrollment Student retention Student graduation Student personal and academic goal attainment Acquisition of a student academic identity Moderate influence Less influence – Multiple external influences

Example: Logic Model Training Program Situation: Funder requires grantees to include a logic model in funding request; grantees have limited understanding of logic models and are unable to fulfill the funding requirement. Inputs Trainer 3 hours training Budget 5 interactive activities Equipment Group work Research base Training curriculum Outcomes Activities & Outputs Practice Increase knowledge of logic models Grantees Increase ability to create a meaningful logic model of program Use logic models in planning and evaluation – in your own work Model quality logic model practice Q and A Increase confidence in using logic models Accountable here Improved planning -programs achieve positive results Improved evaluation more credible and useful data

Anti-Smoking Program Logic External Factors: Tobacco lobby, Lost federal and state tax revenue, Teenage attitudes, Advertisement industry, Price of tobacco, State and local laws Inputs (Resources) Federal funds Staff (FTE) Materials Facilities Volunteers Activities (Processes) Outputs (products) Expect to see Outcomes School-based awareness and prevention seminars Non-smoking incentives Increased awareness of smoking dangers Produce and air TV & radio ads Weekly advertisements and promotions Plan and hold town meetings Bi-monthly meetings Increased number of teenagers in treatment programs Operate nonsmoking treatment clinics Three-phase treatments per clinic (by state) (by region) (by age) (by sex) (by race) Like to see Outcomes Increased number of teenagers completing nonsmoking treatment program (by state) (by region) (by age) (by sex) (by race) Like to see Outcomes Increased number of teenagers and young adults who have remained nonsmokers for at least ten years after quitting (by state) (by region) (by age) (by sex) (by race) End Outcomes All Americans should reach a life expectancy age of 76 years (by state) (by region) (by age) (by sex) (by race)

Potential Evaluative Questions In a Program’s Logical Framework

Action Area 3: Visions & Missions & Goals from different altitudes

Importance of Vision, Mission and Goals (Higher Education) “The most important task, and also the most difficult one, is to define the institution’s mission and develop its goals. That is the first task of the president. Everything else follows from that; everything else will depend upon it. The mission and goals must be ambitious, distinctive, and relevant to the needs and interests of campus constituents. The vision drives the goals, as the president establishes the benchmarks and articulates the values on which the day-to-day life of the institution will depend. Those goals, developed item by item, unit by unit, set the agenda, the blueprint for action, the mandate for change. This, too, is a joint effort: Trustees, provosts, vice presidents, deans, faculty, staff members, students, alumni, the public, advisors, and consultants – all have a role and a proportionate voice – actively influencing decisions, motivating effort, and channeling resources. ” Frank H. T. Rhodes (1998), President of Cornell

State Level

Illinois Board of Higher Education Vision, Mission & Strategic Goals Vision Mission Strategic Goals In an era of rapid economic, technological, and demographic change, the vision of the IBHE is to help build an economically and socially stronger Illinois through increased success in postsecondary degree completion for all residents of Illinois. The IBHE is dedicated to providing greater access and affordability to a quality postsecondary education for all Illinois resident in an effective and efficient manner to meet the needs of the State. Affordability - Ensure that college is affordable for all Illinoisans, particularly low-income students Attainment - Improve educational attainment for all Illinois students, through a seamless P-20 system of high quality teaching and learning, through an increased focus and outreach to nontraditional student, and through stronger emphasis on preparing graduates in high-demand workforce areas Access & Diversity - Increase access and success in a more diverse student body and

Illinois Community College District Level

Illinois Community College System - Vision (Promise For Illinois), Mission & Strategic Goals (Pledges) - Vision Promise for Illinois needs a citizenry composed of individuals who can read, write, compute, communicate, and contribute to society as a whole, as well as to local and global competitiveness. To accomplish this, it need high-quality, affordable, and accessible education. It needs an education system that accommodates smooth transition for students as they move among and between levels of education and work. This educational system must assist them to be successful in reaching their goals. Mission The mission of the ICCB, as the state coordinating board, is to administer the Public Community College Act in a manner that maximizes the ability of the community colleges to serve their communities, promotes cooperation within the system, and accommodates those State of Illinois initiatives that are appropriate for community colleges. Community colleges are committed to providing high-quality, accessible, and cost-effective educational opportunities for the individuals and communities they serve. Strategic Goals “Pledges” High Quality - Emphasize high quality in all programs, services, and operations Affordable Access – Deliver affordable and accessible learning opportunities for all residents of Illinois. Competitive Workforce – Address workforce and economic development needs with flexible, responsive, and progressive programs and services Effective Transitions – Offer rigorous courses, programs, and services designed to enable students to transition from one learning environment and level to another. Enhanced Adult Education – Enhance adult education and literacy programs necessary for individuals and families to achieve high-quality standards of living in Illinois. Service for Student Success – Provide programs and services to assist students succeed in their educational endeavors.

Community College Program Designs

Lincoln Land Community College - Vision, Mission & Strategic Goals Vision Mission Strategic Goals LLCC aspires to be a diverse, learner-centered institution that is a recognized leader, anticipating and responding to academic, economic and cultural needs of the community and assisting individuals to realize lifelong educational goals and reach their full potential. To provide district residents with quality educational programs and services that are accessible, affordable and responsive to individual and community needs. Student Access and success Financial strength Economic responsiveness Community engagement Diversity & cultural competency Operational strength

Program Level - Vision, Mission & Strategic Goals Vision Mission Strategic Goals

Alignment of Vision, Mission & Strategic Goals So very important!

Review the Plausibility of the Program Handout: Guidelines for Reviewing the logic and plausibility of a program’s theory

Action Area 4: Your valued future - evidence-based -

- Focus It is crucial to have a focus for organizing evaluative criteria & identifying sources of information. As Yogi Berra said, “if you do not know where you are going, you will never know when you get there. ” Thus, know what the big picture evaluation question is before you do anything: …on process evaluation (e. g. , quality of design, delivery) …on outcome evaluation (e. g. , values of outcomes) …on learnings (e. g. , barriers & enablers) …on forward thinking (e. g. , exportability)

Use Theory of Change Elements Community Problem/Need Data/Statistics documenting the need Specific Intervention Intended Outcome Evidence • Guides choice of intervention • Supports program improvement

- Quick Quiz – A Simple Logical Framework RESOURCES PROGRAM Expect to see (Outcomes) Like to see (Outcomes) Love to see (Outcomes) The ingredients you need to implement a program What you do in hopes of achieving identified goals or addressing needs Changes in: (knowledge) (skills) (attitude) (motivation) (opinions) Change in behavior, e. g. , improved functioning or performance Change in conditions, e. g. , (social) (educational) (economic) (environmental ) Inputs Activities Means 1. 2. 3. 4. Ends Where would one look to determine program need? Program value? Where would one look to determine underlying needs of program recipients and whether those needs were being addressed? Where would one look for information to determine how to improve the program? Where would one look to determine whether the program was worthwhile, i. e. , worth the costs?

Performance Measures Outcome Chain of Evidence Expect To See Outcome Like To See Outcome Love To See Outcome Students learn job seeking skills Students go on job interviews Student graduates obtain full-time paid employment Indicators: Indicator: # and % of students who can meet criteria in mock interviews # and % of students who go on at least 2 job interviews # and % of graduates who have full-time paid employment # and % of students who develop a resume # and % of employees providing positive feedback about job candidates’ interview skills

UIC CHANCE Program for Underrepresented Students Spheres of Program Influence on Achieving Outcomes P R O G R A M R E S O U R C E S & S T R A T E G I E S Expect To See Outcomes Changes in behavior, relationships, & attitudes of underrepresented students • Increased awareness of the UIC undergraduate experience by students in the CPS and CCC • Increased enrollment of Outputs underrepresented students (e. g. , incoming freshmen and transfer What students) engage in & receive from UIC CHANCE • Students learn new skills (e. g. , technology, study skills) for persisting in Program their academic program Direct control • Students experience increased satisfaction and comfort in the UIC community • Increased number of students participating in health & wellness programs (e. g. , physical, social, emotional, environmental, etc. ) • Students increase their awareness of cultural differences and social justice & equity issues • Increased awareness and engagement in leadership development and community service • Students increase their awareness and knowledge of civic & professional ethics Strong influence Like To See Outcomes Love To See Outcome Changes in status of UIC underrepresented students High numbers of underrepresented UIC students graduating with a foundation of knowledge, skills, attributes and commitment necessary for employment, leadership and citizenship in an increasingly diverse, technologically sophisticated, and complex global society • • • Increased enrollment Student retention Student graduation Student personal and academic goal attainment Acquisition of a student academic identity Moderate influence Less influence – Multiple external influences

All performance measures that have ever existed for any program in the history of the universe involve answering two sets of interlocking questions. Friedman, M. (2007). Trying hard is not good enough: How to produce measurable Improvements for customers and communities. Trafford Publishing

First Set Quantity Friedman, M. (2007) Quality How much How well did we do? did we do it?

Second Set Effort How hard did we try? Effect Is anyone better off? Friedman, M. (2007)

Four Quadrants of Program Performance Measures Effect Effort Quantity Quality How well What and how did we much did we deliver/do it? provide/do? How hard did we try? How much change / effect did we bring about? What is the quality of change / effect we brought about? Is anyone better off? Friedman, M. (2007)

Example: Liberal Arts Program Quantity Quality Effort How hard did we try? Variety of course offerings. Student-Faculty relationships Number of students recruited Were the offerings/experiences relevant & meaningful? Effect Is anyone better off? Number of students retained from full/first time to subsequent 3 rd semester % and type of students entering into four year institutions majoring in liberal arts

A Call to Action Discovery Questions - Checklist