Strategies for Involving Stakeholders in Developing Evaluation Questions

Strategies for Involving Stakeholders in Developing Evaluation Questions Hallie Preskill, Executive Director, Strategic Learning and Evaluation Center hallie. preskill@fsg-impact. org Presented at the American Evaluation Association Annual Conference November 13, 2010 www. fsg-impact. org Boston Geneva San Francisco Seattle

The Evaluation Process Stakeholders & Evaluation Questions Articulate Theory of Change & Develop Program Logic Model 3 Focus the Evaluation Determine Design and Data Collection Methods Design 2 Communicating and Reporting 6 Develop Recommendation s and Action Plans 5 Analyze Data and Interpret Findings 4 Collect Data Implementation Refine Strategy Based on Evaluation Findings 1 © FSG Social Impact Advisors

Evaluation Stakeholders One who has substantial ego, credibility, power, futures, or other capital invested in the program, and thus can be held to be to some degree of risk with it. This includes program staff and many who are not actively involved in the day-to-day operations-for example, inventors or instigators or supporters of the program. (Scriven, 1991) Evaluation stakeholders are those who have a vested interest in the evaluand, and thus would be in a position to use the evaluation results in some way. © FSG Social Impact Advisors

Types of Stakeholders ü ü ü ü ü Program staff Students Clients/customers Program designers Funders Parents Program delivery staff Managers Executives Administrators ü ü ü ü ü Donors Community leaders Elders Community members Constituents Policy makers Teachers/educators Legislators Policy makers Researchers 4 © FSG Social Impact Advisors

Involving Stakeholders in an Evaluation • Increases quality, scope, and depth of questions • Ensures transparency • Facilitates the evaluation process • Acknowledges political context of evaluation • Builds evaluation capacity • Fosters relationships and collaboration © FSG Social Impact Advisors

Relationship between Evaluation Questions and Findings © FSG Social Impact Advisors

Good evaluation questions… • Establish the boundary and scope of an evaluation, and communicate to others what the evaluation will and will not address • Are the broad, overarching questions that the evaluation will seek to answer; they are not survey or interview questions • Are aligned with clearly articulated goals and objectives • Can be answered through data collection and analysis ***Reflect diverse perspectives and experiences*** © FSG Social Impact Advisors

5 -Step Process for Engaging Stakeholders Step 1: Prepare for stakeholder engagement Step 2: Identify potential stakeholders Step 3: Prioritize the list of stakeholders Step 4: Consider potential stakeholders’ motivations for participating Step 5: Select a stakeholder engagement strategy © FSG Social Impact Advisors

Step 1: Prepare for stakeholder engagement This step includes identifying: • When the program started and who sponsored its development • The underlying assumptions about why the program exists • The resources allocated to the program • The activities the program undertakes to achieve its goals • The expected outputs and short- and long-term goals or outcomes • The external forces that have affected or currently influence the program’s design and implementation © FSG Social Impact Advisors

Step 2: Identify potential stakeholders Potential stakeholders can be identified by what they would contribute to the evaluation process © FSG Social Impact Advisors

Step 3: Prioritize the list of stakeholders Stakeholders can be prioritized by whether they are: • Vital to the evaluation’s success and resulting use of findings • Important to include for practical or political reasons • Nice to include if possible given time and resources © FSG Social Impact Advisors

Step 4: Consider stakeholders’ motivations for participating Stakeholder motivations might include: • Commitment to the goals of the program • Personal stake in the program • Professional development • Opportunity to earn additional income © FSG Social Impact Advisors

Step 5: Select a stakeholder engagement strategy An appropriate strategy can be affected by: • Time available • Budget • Geographic locations of the stakeholders • Range of stakeholder perspectives • Extent to which stakeholders have existing relationships • Stakeholders’ availability to engage • Number of stakeholders • Extent to which the stakeholders are familiar with evaluation • The degree of program complexity © FSG Social Impact Advisors

Stakeholder Engagement Strategies and Criteria © FSG Social Impact Advisors

Stakeholder Engagement Techniques © FSG Social Impact Advisors

Challenges to Engaging Stakeholders in the Evaluation Question Development Process Time limitations Differing opinions, priorities, and information needs Stakeholder tensions Differing knowledge of evaluation Stakeholder indifference Stakeholder accessibility © FSG Social Impact Advisors

© FSG Social Impact Advisors