Representing Simple Complicated and Complex Aspects in Logic

Representing Simple, Complicated and Complex Aspects in Logic Models for Evaluation Quality Presentation to the American Evaluation Association conference, San Antonio, TX November 2010 Patricia Rogers Patricia. Rogers@rmit. edu. au

Truth • The truth • The whole truth • And nothing but the truth 2

Beauty • “walking in beauty”/ harmony 3

Justice • Without fear or favor • Balancing the scales 4

5

The material in this presentation is drawn from Purposeful Program Theory: Effective Use of Theories of Change and Logic Models, by Sue C. Funnell and Patricia J. Rogers , ISBN: 9780470478578, John Wiley/Jossey-Bass, February 2011. Funnell and Rogers Purposeful Program Theory 6

What I mean by terms used today Program theory An explicit theory of how an intervention contributes to the intended or observed outcomes, which has 2 components: Theory of change The process by which change comes about (for an individual, organization or community) Theory of action How the intervention is constructed to activate theory of change Logic model A visual representation of a program theory, usually in a diagram Program theory evaluation An evaluation that is at least partly guided by an explicit program theory. It is not necessarily ‘driven’ by theory, since it should be driven by its intended purpose and the needs of its intended users Funnell and Rogers Purposeful Program Theory 7

Aspects of complicated and complex situations and interventions with potentially important implications for evaluation 1) Focus 2) Governance 3) Consistency 4) Necessariness 5) Sufficiency 6) Change trajectory 7) Unintended outcomes Funnell and Rogers Purposeful Program Theory 8

Simple interventions Funnell and Rogers Purposeful Program Theory 9

Complicated interventions Intervention that produces different outcomes valued by different stakeholders Funnell and Rogers Purposeful Program Theory 10

Focus –Complicated - Objectives at multiple levels of a system Funnell and Rogers Purposeful Program Theory 11

Complex interventions Focus – Complex - Emergent outcomes Funnell and Rogers Purposeful Program Theory 12

(2) Governance Simple Single organization Complicated Specific organizations with formalized requirements Complex Emergent organizations working together in flexible ways Funnell and Rogers Purposeful Program Theory 13

(4) Necessariness Simple Only way to achieve the intended impacts Complicated One of several ways to achieve the intended impacts – which can be identified in advance Complex One of several ways to achieve the intended impacts – which are only evident in retrospect Funnell and Rogers Purposeful Program Theory 14

(5) Sufficiency Simple Sufficient to produce the intended impacts. Works the same for everyone Complicated Only works in conjunction with other interventions (previously, concurrently, or subsequently) and/or only works for some people and/or only works in some circumstances – which can be identified in advance Complex Only works in conjunction with other interventions (previously, concurrently, or subsequently) and/or only works for some people and/or only works in some circumstances – which is only evident in retrospect Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass) Funnell and Rogers Purposeful Program Theory 15

Ways in which an intervention can work with other interventions Stronger Families and Communities Strategy evaluation 2000 -2004 Final Report http: //www. rmit. edu. au/casr/sfcse Funnell and Rogers Purposeful Program Theory 16

Sufficiency – Complicated - Multi- stage intervention Funnell and Rogers Purposeful Program Theory 17

Sufficiency – Complicated – Intervention that works in combination with other interventions Funnell and Rogers Purposeful Program Theory 18

Sufficiency – complicated - intervention that works differently for different types of participants Funnell and Rogers Purposeful Program Theory 19

Change trajectory – complex - Tipping points Funnell and Rogers Purposeful Program Theory 20

(7) Unintended outcomes Simple Unintended outcomes can be anticipated and monitored Complicated Different unintended outcomes are likely in particular combinations of circumstances – expertise is needed to anticipate them and identify them Complex Unintended outcomes cannot be anticipated but only identified (and addressed) as they emerge or in retrospect Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass) Funnell and Rogers Purposeful Program Theory 21

Generic theory of change with emergent theory of action Funnell and Rogers Purposeful Program Theory 22

What people sometimes assume you mean by logic models that address complexity Funnell and Rogers Purposeful Program Theory 23

Two framings of simple, complicated and complex Simple Glouberman and Zimmerman 2002 Kurtz and Snowden 2003 Tested ‘recipes’ assure replicability The domain of the ‘known’, Expertise is not needed Cause and effect are well understood, Best practices can be confidently recommended, Complicated Complex Success requires high level of expertise in many specialized fields + coordination The domain of the ‘knowable’ Every situation is unique – previous success does not guarantee success The domain of the ‘unknowable’, Expert knowledge is required, Patterns are only evident in retrospect. Expertise can help but is not sufficient; relationships are key Kurtz, C. F. and D. J. Snowden (2003) ‘The New Dynamics of Strategy: Sense-making in a Complex and Complicated World’, IBM Systems Journal 42(3): 462– 83. Funnell and Rogers Purposeful Program Theory 24

Some thoughts on how evaluation might help us to understand the complicated and the complex Issues that may need to Possible evaluation methods, approaches and methodologies be addressed 1. 2. 3. 4. 5. 6. 7. Focus Governance Consistency Necessariness Sufficiency Change trajectory Unintended outcomes • Emergent evaluation design that can accommodate emergent program objectives and emergent evaluation issues • Collaborative evaluation across different stakeholders and organisations • Non-experimental approaches to causal attribution/contribution that don’t rely on a standardized ‘treatment’ • Realist evaluation that pays attention to the contexts in which causal mechanisms operate • Realist synthesis that can integrate diverse evidence (including credible single case studies) in different contexts • ‘Butterfly nets’ to catch unanticipated results Funnell and Rogers Purposeful Program Theory 25

Resources and references Glouberman, S. , and Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Ottawa: Commission on the Future of Health Care in Canada, 2002. http: //www. healthandeverything. org/fi les/Glouberman_E. pdf. Kurtz, C. F. , and Snowden, D. F. “The New Dynamics of Strategy: Sense. Making in a Complex and Complicated World. ” IBM Systems Journal, 2003, 42(3), 462– 483. Rogers, P. J. “Using Programme Theory for Complicated and Complex Programmes. ” Evaluation, 2008, 14 (1), 29– 48. Rogers, P. J. , Guijt, I. , and Williams, B. “Thinking Systemically: Seeing from Simple to Complex in Impact Evaluation. ” Presented at the 3 IE/African Evaluation Association Impact Evaluation Conference, Cairo, Egypt, 2009. Snowden, D. J. , and Boone, M. “A Leader’s Framework for Decision Making. ” Harvard Business Review, Nov. 2007, pp. 69– 76. 26