A Tool for Designing Evaluations of Paradigm Shifts

Session Objective § To build capacity of evaluators to design evaluations of interventions in

Four Phases of Evaluation Design Evaluation Shape Practice Collect Data Make Meaning from Data

Tool’s Role in Evaluation Design § Tool represents the intervention’s theory of change and

Session Agenda § Introduction to tool for designing evaluations of interventions in complex systems

Theory of Change in Paradigms, Structures, and Conditions of Complex Systems Example from Cross-Site

Tool Characteristics § Tool focuses on deep patterns of change in systems (i. e.

Differences from Typical Logic Model § Will almost always depict more aspects of the

What is a System? § A group of interacting, interrelated, and interdependent elements forming

The Systems Iceberg Events and Behaviors Patterns Structures Paradigms Conditions What is happening now?

Deep System Change § Deep system change involves shifts in paradigms and structures of

Paradigm Shift: Risk to Protective Factors Risk Factors • Substance Abuse • Poverty •

Dynamics of a Social System and Its Context Agreement Context Low Unorganized High Agreement

Group Task § Succinctly describe the paradigm shift. § Identify the ultimate beneficiary of

Phases of Shift § Baseline understanding of fundamentals and system dynamics § § Trying

Other Key Evaluation Design Questions § Is the evaluation internal or external to the

Slides: 22

Download presentation

A Tool for Designing Evaluations of Paradigm Shifts in Complex System Interventions Beverly Parsons In. Sites www. insites. org bparsons@insites. org Marah Moore I 2 i Institute www. insites. org marah@i 2 i-institute. com Pat Jessup In. Sites www. insites. org pjessup@insites. org Handouts and Power. Point presentation available at: www. insites. org/AEA 2010 1

Session Objective § To build capacity of evaluators to design evaluations of interventions in complex systems 2

Four Phases of Evaluation Design Evaluation Shape Practice Collect Data Make Meaning from Data 3

Tool’s Role in Evaluation Design § Tool represents the intervention’s theory of change and relationship of intervention to larger system/situation § Evaluation designers also need to know: – Whether evaluation is part of intervention (internal) or is separate from intervention (external) – If evaluation is part of intervention, does it participate through • Evaluation processes and/or • Evaluation findings 4

Session Agenda § Introduction to tool for designing evaluations of interventions in complex systems § Work in small groups to begin tool development § Reconvene to discuss completion of tool and its use 5

The Tool 6

Theory of Change in Paradigms, Structures, and Conditions of Complex Systems Example from Cross-Site Evaluation of Quality Improvement Center on Early Childhood 7

Tool Characteristics § Tool focuses on deep patterns of change in systems (i. e. , changes in structures and paradigms) § Tool recognizes multiple system dynamics at play in complex systems 8

Differences from Typical Logic Model § Will almost always depict more aspects of the system than just the intervention § Depicts phases of change over time with variation in rates and patterns across components of the system § Likely to depict the intervention through a different lens 9

Background Concepts 10

What is a System? § A group of interacting, interrelated, and interdependent elements forming a complex whole § A configuration of parts connected and joined together by a web of relationships § The whole is greater than (different than) the sum of its parts 11

The Systems Iceberg Events and Behaviors Patterns Structures Paradigms Conditions What is happening now? How do patterns play out over time and space? What are the drivers? How are they related? 12

Deep System Change § Deep system change involves shifts in paradigms and structures of system boundaries, relationships, perspectives, and dynamics over time and space. § Change is evident in mid-level patterns through data analysis/synthesis. § Change is evident in observable events and behaviors. 13

Paradigm Shift: Risk to Protective Factors Risk Factors • Substance Abuse • Poverty • Stress Protective Factors • Parental resilience • Social connections • Concrete support in time of need • Knowledge of parenting and child development • Social and emotional competence of child 14

Dynamics of a Social System and Its Context Agreement Context Low Unorganized High Agreement Perspectives (random) Self. Organizing (complex, adaptive) Organized (simple, complicated) High Predictability Relationships Low Predictability 15

Group Task § Succinctly describe the paradigm shift. § Identify the ultimate beneficiary of the paradigm shift. § Identify the likely key points of leverage to shift the system. § Order the likely points of leverage along left side of page beginning with the ultimate beneficiaries of the system and those close by. 16

Phases of Shift § Baseline understanding of fundamentals and system dynamics § § Trying out interventions in system § Sustainable adaptive balance of new fundamentals and system dynamics in a shifting context Tipping point to new fundamentals and system dynamics balance 17

Theory of Change in Paradigms, Structures, and Conditions of Complex Systems Example from Cross-Site Evaluation of Quality Improvement Center on Early Childhood 18

Other Key Evaluation Design Questions § Is the evaluation internal or external to the intervention? § What are the boundaries of the intervention? § What are the focal points of the evaluation? § How will the evaluation attend to the larger system (beyond the intervention)? 19

Example of Evaluation Design 20

QIC-EC Cross-Site Analysis Framework 21