AEA Conference 2014 Denver Systems Thinking Learning and

AEA Conference 2014, Denver Systems Thinking, Learning and Values in Evaluation October 16, 2014 Session: Applying Systems Thinking to Peacebuilding, Homelessness and Values Richard Hummelbrunner ÖAR Regionalberatung Graz, Austria

The Issue. . . • Learning is a purpose postulated for most evaluations, but Ø main specifications: what is to be learned, who should do the learning or who should be involved Ø the envisaged type of learning is rarely specified, remains vague • Questions of values are clearly important in evaluations, but Ø values underpinning an evaluation are often vague or implicit Ø can partly be deducted from the approach or methodology applied Learning and values usually dealt with separately in evaluations, although they mutually influence each other or can be seen as complimentary - thus should be viewed

Types of Learning (from Organisational Development) Effects Are we doing things right? Are we doing the right things? How do we establish ‘rightness’? Single loop learning Double loop learning Triple loop learning Action Norms, rules, assumptions Change actions Change generative structure Change learning mechanisms Cognition- / behaviour patterns

Core Systems Concepts • • • Interrelationships Ø Connections of elements in a system (particularly dynamic and non-linear aspects, feedback) Perspectives Ø Perception of connections is framed by Stakeholders (roles) and Stakes (values and motivations) Boundaries Ø Critical awareness of the need to draw boundaries, reflect on the consequences of boundary choices (incl. power issues) Generic concepts as common denominators across the systems field (wide range of approaches, methods)

Types of Learning and Systems Concepts Triple loop learning‘How do we establish ‘rightness’? Double loop learning Are we doing the right things? Are we doing Single loop learning things right? Intervention Effects Interrelationships Perspectives Boundarie s Questions purpose, goals Power for determining purpose and goals

Types of Value (in applied social science) • Instrumental value Ø value as a means to promote some ends (e. g. money) Ø once it ceases to be a means it has no value • Intrinsic (or non-instrumental) value Ø value as an end-in-itself regardless of whether it is a means to promote anything else (e. g. objects in nature) • Critical value Ø closely associated with issues of power, expertise and legitimacy (e. g. democratic values, social justice, equity)

Types of Learning and Values Triple loop learning‘How do we establish ‘rightness’? Double loop learning Are we doing the right things? Are we doing Single loop learning things right? Intervention Effects Instrumental values Efficiency Intrinsic values Critical values Relevance, effectiveness Equity, sustainability

Boundaries Perspectives TY VA PES LU O ES F SYSTEMS CONCEPTS Three-dimensional framework for coherence Critical Intrinsic Interrelationships Instrumental Single. Loop Double. Loop Triple. Loop TYPES OF LEARNING

Possible uses of proposed framework • • • Interrogate coherence among components of an evaluation Ø Implications of the location of a particular typology (e. g. learning) for application of values or systems concepts Align approaches to the situation (simple, complicated, complex) Ø Select appropriate types of learning, values or systems concepts Reflect on implications of impact evaluation designs Ø Appropriate matching for a chosen design (e. g. experimental designs, theory- or case-based approaches, participatory)

Utility of proposed framework • • Achieve more coherent use of learning, values and systems concepts in evaluation assignments Improve the rigor of evaluation designs Ø not by privileging certain methods, but by emphasizing coherence and congruence between three important aspects. Achieve a better fit between mental maps Ø espoused theories: declared claims (explicit) Ø theories in use: what is actually done (often implicit) Framework applicable for various evaluation stakeholders (e. g. commissioners, evaluators) in order to Ø reflect on the constraints and limitations of an evaluation Ø point at hidden opportunities that might be missed

Further reading • • • Argyris, C. and D. Schön (1996): Organizational Learning II. Theory, Method and Practice. London, Addison-Wesley. Ryan, K. E. and De. Stefano, L. (eds): Evaluation as a Democratic Process: Promoting inclusion, dialogue, and deliberation. New Directions for Evaluation, No. 85 (pp. 3 – 12). Jossey-Bass. Kimmel, Allen J (1988) Ethics and Values in Applied Social Research Methods Series 12. Sage. Hummelbrunner Richard (forthcoming): Learning, Systems Concepts and Values in Evaluation: Proposal for an exploratory framework to improve coherence. IDS Bulletin 4. 9, Brighton. Rogers, Patricia (2008). Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation 14 (1): 29– 48.

Thank you for your attention! Richard Hummelbrunner ÖAR - Regionalberatung Baumgasse 10, 8045 Graz, AUSTRIA T: +43/664 255 6208 hummelbrunner@oear. at www. oear. at