Making Contribution Analysis Work The Benefits of Using

Making Contribution Analysis Work The Benefits of Using Contribution Analysis The 25 th Annual Conference of the American Evaluation Association

Contribution Analysis – what is it? John Mayne (2001): “[a] specific analysis undertaken to provide information on the contribution of a program to the outcomes it is trying to influence”

Why Contribution Analysis? • The world is complex and messy • Counterfactual designs are not always possible or desirable (Cook et al. , 2010) • Different types of information are needed

What is the aim of contribution analysis? To provide a an alternative and noncounterfactual way to address the attribution challenge To provide the best possible evidence that the intervention contributed to the outcome of interest

What are the key steps of CA? Step 1. Setting out the cause-effect issue to be addressed Step 2. Developing the postulated theory of change (embedded) Step 3. Gathering the existing evidence on theory of change Step 4. Assembling and assessing the contribution story (and the challenges) Step 5. Seeking out additional evidence Step 6. Revising and strengthening the contribution story

The embedded theory of change

What Does a Good CA Look Like? The Embedded Theory of Change contains: A. The Implementation theory B. The underlying assumptions and mechanisms C. Positive and negative side effects D. Influencing factors E. Alternative explanations

The strengths of CA • Transferability of knowledge • Feasibility • Handles complexity (thick description) • Stakeholder involvement

Theory-based evaluation focuses on contribution in causal links § TBEs typical challenges § Focuses solely on the implementation theory § Too simplistic § The theory of change is not used to guide the evaluation § Minimal attention paid to alternative explanations

Strengths continued ü The model uses the embedded theory of change as guideline for structuring all phases in the evaluation ü It illustrates and documents each step and hypothesis in the embedded theory of change ü Makes contribution probable and creates knowledge of effective mechanisms in the intervention ü Accounts for alternative explanations ü Accounts for implementation factors ü Provide the opportunity for distinction between implementation error and theory error

Still some holes to cover! How do we account for alternative explanations and external factors? How do we carve out time and resources for the reiterative process in commissioned work

The Benefits of Using Contribution Analysis in Public Sector Settings

Agenda • Use in (conventional) evaluation • Use in public management steve. montague@pmn. net www. pmn. net 13

Use in Evaluation • Much discussed – little used • Challenges – ‘Sloppy’ results logic – Methods driven ideas of rigour – Fear of ‘exposure’ steve. montague@pmn. net www. pmn. net 14

Original Sector ‘X’ Market Access / Development Program Ministry (Funding Agency) Activities Conduct analysis to address key science gaps Science-based information products Outputs Industry Association Conduct market research by monitoring and reporting on market trends Environmental scans, surveys & market intelligence products Annual work plans Conduct market outreach and leadership activities in key markets Market outreach communication tools and products [tailored for target audiences] Tours, meetings and workshops Networks of experts and stakeholders Canadian sector ’x’ is well-informed of potential barrier issues and trends in international markets Immediate Outcomes Intermediate Outcomes Communicate science-based evidence to Canadian stakeholders Science-based evidence supports sector ‘x’ market outreach Improved coordination of sector ‘x’ market advocacy efforts in a way that responds to market realities Key influencers have the information they need regarding the (barrier related) credentials of Canada’s sector ‘x’ and its products The Canadian sector ‘x’ industry works proactively with policy-makers in target markets to ensure that decisions are based on scientific evidence Canadian sector ‘x’ products are considered to be an appropriate (re: potential barrier) responsible & preferred choice internationally Reduce and avoid market access issues in international target markets End Outcome steve. montague@pmn. n et Improved acceptance for Canadian sector ‘x’ products in international markets www. pmn. net 15

Alternative Results Chain for Sector X Market Access / Development Results Chain I MP L E M E N T A T I O N 1. Government Department (funding agency) determines n eed. 2. Government Department (funding agency) invests in program(s). 3. The appropriate governance arrangements and (critical mass of) co-delivery agents engage with Government Department and other ‘partners’. Assumptions / External Factors A. Appropriate information, understanding and analysis of problems convert into appropriate investment B. Sufficient, appropriate and consistent funding and program assistance C. Agendas remain consistent with key co-deliverers 4. Governance structures are formed and actively managed (Advisory Committees and contracted deliverers). D. Support climate allows for clear governance 5. Priorities are (clearly) set and projects are solicited (appropriately). E. Economic, management and political circumstances allow for appropriate sector engagement 6. Appropriate industry sector participation / engagement in project selection. 7. Appropriately targeted and realistic proposals supported (i. e. they respond to market realities). F. Key sector proponents have the capacity and commitment to apply for targeted assistance G. Proponents have ‘will’ and ability to carry through on commitments 8. Projects are conducted as anticipated (appropriately addressing needs). Sector ‘x’ industry works with policy makers to encourage use of scientific evidence in decisions 9. Appropriate target groups (e. g. market acceptance and access community) are sufficiently ‘reached’ / engaged by market development initiatives / projects. C H A N GE 10. Groups reached by initiatives show positive reactions, capacity (knowledge, abilities, commitments, aspirations) - willingness and commitment to using scientific evidence in decisions, key influencers have info they need re: sector ‘x’ and products. Canadian sector ‘x’ products are considered to be appropriate (vis a vis potential barrier) responsible and preferred choice internationally. 11. Incremental change to policies and practices related to the use of sector ‘x’ products (empirically focussed, expanded and improved) - reduced access issues Improved acceptance for Canadian sector ‘x’ products in international markets. Market access issues are reduced and avoided in international target markets. H. Target communities attracted to participate / engage in initiatives (for the right reasons) I. Messages / information / supports are ‘attractive’ and compelling to participants J. Groups have broad economic, policy and management support and conditions K. Canadian sector ‘x’ products would not be sold elsewhere 12. Improved ‘acceptance’ of Canadian sector ‘x’ products. Increased sales of Canadian products. L. Canadian sector ‘x’ products cost of goods sold allow for a net profit 13. Net benefit to Canadian sector ‘x’ companies. 14. Net benefit to Canada and Canadian communities. steve. montague@pmn. n et M. Net benefits to Canadian sector ‘x’ companies create net benefits for Canadian communities www. pmn. net 16

Evaluation Use: Conclusions • Implementation and Change Theory discussion very useful • ‘Honed’ evidence • No ‘surprises’ when theories laid out early and often • Can actually increase engagement of stakeholders steve. montague@pmn. net www. pmn. net 17

Use in Public Management • Institutional arrangements not conducive to results focus • Simple scorecards – Misleading – Dangerous steve. montague@pmn. net www. pmn. net 18

The Relevant Explanation Finder (REF) A Practical Tool to Examine Alternative Explanations

The Challenge How do we account for alternative explanations and external factors?

Mechanisms – what are they? • Conceptual plurality • Astbury and Leeuw (2010): “underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest” – Not variables! – Not activities

• Mechanisms exhibit the following characteristics: – they are usually hidden; – they are sensitive to variations in context; and – they generate outcomes (Astbury & Leeuw, 2010: p. 368).

Alternative Explanations Alternative explanations are (typically a complex of) mechanisms that are separate from the mechanisms intended to drive the desired outcomes. Two types: - Direct rival - Commingled rival

Influencing Factors Influencing factors are contextual factors that either propel or inhibit mechanisms driving the desired outcomes.

Influencing Factors • The individual capacities of the key actors and stakeholders such as interests, attitudes, capabilities and the credibility of (for instance) professionals or beneficiaries. • The interpersonal relationships required to support the intervention, such as lines of communication, management and administrative support, union agreements and professional contracts. • • • The institutional setting in which the intervention is implemented, such as the culture, leadership etc. of the implementing body. The wider (infra-)structural and welfare system, such as political support, the availability of funding resources etc. (Pawson & Greenhalg, 2004: p. 65)

The REF Descri ption Type Level Identi fiers Certai nty Robus tness Range Preval ence Gener alized specifi city

REF Component Explanation 1. Description 2. Type - Direct rival Commingled rival Implementation rival (factors) 3. Level - Individual Interpersonal Institutional -Infra-structural 4. Identifiers Specific data patterns indicating the presence of rivals 5. Degree of influence 1. 2. 3. 4. 5. 6. Implications Certainty Robustness Range Prevalence Generalized specificity

The Benefits • Allowed us to develop a chain-ofevidence (transparency) • Provided a focusing structure in our data analysis (systematic) • Transferability (critical mechanisms)

The Next Steps … 1. Application in more settings 1. Will the REF demonstrate relative advantages in terms of enhancing the credibility of contribution stories (over an above doing something else, or nothing at all)? 2. What, if any, unintended outcomes result from the practical application of the REF?

Steve Montague (steve. montague@pmn. net) Sebastian Lemire (setl@r-m. com) Steffen Bohni (sni@r-m. com)

References John Mayne (2001). Addressing attribution through contribution analysis: Using performance measures sensibly, The Canadian Journal of Program Evaluation, 16(1), 1 -24. Cook et al. (2010). Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation, 31 (1), 105 -117. Line Dybdal, Steffen Bohni Nielsen & Sebastian Lemire (2011). Contribution Analysis Applied: Reflections on Scope and Methodology. The Canadian Journal of Program Evaluation, 25(2), 29 -57. Astbury, B and Leeuw, F. (2010). Unpacking Black Boxes: Mechanisms and Theory. Building in Evaluation, American Journal of Evaluation, 31(3), 363 -381. Forthcoming special issue on contribution analysis in Evaluation.