Human Systems Dynamics Theory Applied to Evaluation Practice

Human Systems Dynamics Theory Applied to Evaluation Practice American Evaluation Association 2008 Beverly Parsons, Ph. D. In. Sites bparsons@insites. org Meg Hargreaves, Ph. D. Mathematica Policy Research, Inc. mhargreaves@mathematica-mpr. com 1

2

3

4

Introduction to a Systems Perspective In Evaluation This section presents: • • • System definitions System features System characteristics Types of systems Examples of types 5

Systems Definitions Multiple definitions: • A group of interacting, interrelated, or interdependent parts forming a complex whole • A configuration of parts joined together by a web of relationships • The parts form a whole, which is greater than the sum of its parts 6

System Features Systems are as much an “idea” about the real world as a physical description of it: • Boundaries define who or what lies inside or outside the system • Differences among the parts influence the system’s dynamics • Relationships among parts, between parts and whole, and between whole and its environment, are key focus of systems 7

System Characteristics Common patterns, behaviors, and properties: • Patterns – unorganized, or organic (self-organized) • Behaviors – random, simple, complicated, or complex adaptive; linear or nonlinear • Properties – independent, interrelated, or interdependent relationships • Scale – small to large, self-similarity across levels (fractals) 8

System Types Systems can be grouped by their level of complexity or organization: • Random (no system) - unorganized • Simple system - organized • Complicated system – organized • Complex adaptive system – organic 9

Random (Unorganized) • Random, chaotic activity – no pattern • Independent, unconnected parts • No cause-effect relationships – constant chaos and surprise • Turbulence - no equilibrium • Random parts without a system • No leadership - people react blindly • Unknowable 10

Random System Examples • War zone: Civilians caught in crossfire, random flight to escape conflict • Natural disaster: At landfall or in the eye of the storm, residents react instinctively to events • Leadership transitions: During changes in administration old patterns are suspended before new patterns are established 11

Simple System (Organized) • • Stable, static pattern Parts connected in linear ways Predictable cause-effect relationships Set equilibrium System reducible to parts and replicated Directive leadership - designed change Known knowns – answers are evident 12

Simple System Examples • Baking a cake: Follow a recipe to assemble and combine ingredients into a batter that is baked at a pre-set temperature with predictable results • Flu shot clinics: Nurses use consistent procedures to administer the same shots to each person, following a set protocol in assembly-line fashion 13

Complicated (Organized) • • Dynamic pattern of feedback loops Many interrelated parts across subsystems, levels Complex, nonlinear cause-effect relationships Feedback can stabilize equilibrium – thermostat System can be reduced to parts and replicated Multiple answers – investigate options Unknowns become known through expert analysis at multiple levels 14

Complicated System Examples • Space Shuttle Challenger disintegrated (1986) when O-ring failure caused a rocket booster breach, creating flare that damaged external fuel tank, spilling fuel that exploded • In large healthcare institutions, human behaviors are part of wider systems of causality, in which medical errors can occur in organizational and policy contexts that result in long (36 -hour) shifts, large caseloads, and strained staff relations 15

Complex Adaptive System (Organic) • Dynamical patterns – parts adapting to each other and to environment as a whole • Parts are massively entangled, interdependent • Parts self-organize, learn, coevolve organically • Equilibrium in flux - sensitive to initial conditions • System not replicable, can’t repeat past • Emergent change – manage conditions of organic development and experimentation • Unknown unknowns – trial and error 16

Complex Adaptive System Examples • Economic system – interactions of homeowners, mortgage lenders, stock market traders, investors, federal banking institutions, and worried consumers are coevolving into global crisis and recession, despite governments’ interventions • User networks (Diabetes, AA) facilitate exchange of information and advice on care for chronic conditions among participants, learning from each other 17

Background about Systems Theories This section presents: • General systems theory • Cybernetics – systems dynamics • Complex adaptive systems • Implications for evaluation 18

General Systems Theory • Holistic change ideas – ancient Greeks • General systems theory - von Bertalanffy (1930’s); earliest work by Bogdanov (1910) • Open systems – nonrandom elements organized into interacting, interrelated components that seek to survive through interactions with environment • Each system level nested in higher level (cells, organisms, families, organizations, communities, societies) 19

Implications for Evaluation • The whole can enable/constrain parts and the parts can contribute to and/or challenge stability of the whole • Because open systems are structured in hierarchies; useful to look one level above and one level below the ‘system in focus’ • Evaluate system viability – does system have both the parts and the information and decision flows among the parts that are needed to survive? 20

Cybernetics and System Dynamics • System dynamics founded by Forrester at MIT (1950’s) for electrical engineering • Method for calculating and modeling how many circular, interlocking, sometimes timedelayed relationships among parts are important in shaping system-wide behavior • Through negative feedback, adjustments made to keep system in balance; positive feedback used to move system in same direction, moving out of balance 21

Implications for Evaluation • Assess influence of feedback loops on behavior of system’s parts and on whole • Behavior of whole not only explained by behavior of parts (e. g. medical errors) • Feedback loops undermine sustainability of public interventions (policy resistance) • Evaluators cannot step outside social and ecological systems to observe (not valueneutral); self-reflection needed 22

Complex Adaptive Systems • Key CAS writers – Weaver (1948), Simon (1962), Prigogine (1987), Stacey (1997, 2007), Zimmerman et al (2001), Eoyang (2006) • CAS – many semi-independent and diverse agents, who are free to act in unpredictable ways, continually interact with each other, adapting to each other and to environment as a whole, creating system-wide patterns • Key concepts – emergence, organic selforganization, co-evolution, simple rules 23

Implications for Evaluation • Currently relevant evaluation criteria and measures may need to be updated as new conditions emerge • Measure frequently for emerging patterns • Avoid grand modeling projects for prediction; use smaller projects for ongoing experimentation and learning • Also visualize system interactions as networks; look outside nested levels for system patterns, drivers, and constraints • Ask what, so what, now what? 24

Three Dynamics of a Social System and its Context far from agreement C O N T E X T Unorganized dynamics (random unpatterned seemingly chaotic) close to agreement Agreement organic dynamics (emerging patterns coherent but not predictable) Organized dynamics (predictable orderly controlled) close to certainty far from certainty Certainty 25

Match of Evaluation Designs to Dynamics of Social Systems and Their Context Exploratory Design unorganized dynamic Agreement close to agreement Initiative Renewal Design far from agreement CON T E X T Organic Design organic dynamic Predictive Design organized dynamic close to certainty Certainty far from certainty 26

Understanding Organic Dynamics (Activity) • Divide into triads • Selects one other triad member (doesn’t tell) and uninvolved person in refreshment area • Stay at least two feet apart and equidistant from the other two • Do this for about 1 -2 minutes while trying to reach refreshments • Reflect on experience 27

Case Study Introduction Do the preconference professional development offerings contribute to effective evaluation-related work of association members? If so, how? 28

Unorganized System Dynamics What is happening? What boundaries, differences, similarities, and relationships might shape how the offerings contribute to participants’ evaluation-related work? 29

Organized System Dynamics Do participants receive high-quality content that is relevant to their evaluation -related work and is delivered through high -quality instructional methods? 30

Organized System Dynamics How do the format and content of the session support or hinder participants in understanding and using the session to apply the learning from the session to their evaluation work? 31

Organic System Dynamics What patterns among participants (including the session facilitators) before and during the session are likely to affect the participants’ understanding and application of the learning to their evaluation-related work? 32

Patterns 33

Patterns 34

Patterns 35

Patterns 36

Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY 2005, FY 2006, and FY 2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 10) 37

Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY 2005, FY 2006, and FY 2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 27) 38

Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY 2005, FY 2006, and FY 2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 42) 39

Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY 2005, FY 2006, and FY 2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 78) 40

Patterns 41

Fractals: Patterns, Patterns Everywhere In nature. . . • Mathematical construct of iterating nonlinear equation and plotting on complex number plane—Mandelbrot Set • Similar shapes at all scales—Broccoli • Biological scaling gives coherence in widely diverse entities— Oak tree • Scale-free networks 42

Fractals: Patterns, Patterns Everywhere • Recognizing patterns is critical: similarities, differences, and relationships that have meaning across space and time • Basic values or simple rules generate diverse, but self-similar behavior across scales • Naming and telling stories about dynamics in a system help reinforce and shape fractal patterns 43

Fractals 44

Looking at the Dynamics as a Whole • What is the overall picture of system dynamics affecting how the preconference professional development offerings contribute to effective evaluation-related activities of AEA members? • Given the findings from the three system dynamics within the preconference session, how might the preconference professional development process and offerings be modified to contribute more substantially to the quality of AEA members’ evaluation-related work? 45