How complexity influences evaluation A presentation to the

How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie Mc. Geary

Presentation outline • • • Complex systems Complex interventions Role of program theory in complex evaluations Audience expectations Future Farming Strategy evaluation planning

Complex systems • • • Non linear Emergent Dynamical Adaptive Uncertain Co-evolutionary Patton, M. (2010). Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, NY, p. 8.

Victorian agriculture system

Complex interventions • • • Governance and implementation Simultaneous causal strands Alternative causal strands Non-linearity and disproportionate outcomes Emergent outcomes Rogers, P. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1), p. 32.

The Future Farming Strategy (FFS)

Interactions Complex interventions are events in systems. It is not the intervention itself but the context or setting into which it is introduced and with which it interacts which is the most significant aspect of the complexity. Hawe, P. , Shiell, A. and Riley, T. (2009). Theorising interventions as events in systems. American Journal of Community Psychology, 43, 267 -276.

Role of program theory

Role of program theory Inputs Outputs Transformational potential

Future Farming Strategy logic model

Audience expectations • Judgements of adequacy • ‘Global’ net benefits • Causal inference – level of confidence • Quantification “The tension between the systematic evaluation approaches proposed by evaluation experts and the impact measurements required by clients for accountability purposes is not going to be solved easily. ” Molas-Gallart, J. and Davies, A. (2006). Toward theory-led evaluation: The experience of European science, technology, and innovation policies. American Journal of Evaluation, 27(1), p. 78.

Evaluation planning Purpose • Examine if FFS has made progress towards improving the productive, competitive and sustainable status of Victorian farm businesses • Assess the benefits of Victorian Government investment in the FFS • Provide opportunities for learning and improvement in whole -of-government strategy implementation and delivery to guide future policy and program development.

Evaluation planning Implications of complexity • Outcomes definition • Alignment between Action and strategy level outcomes • Forms of evidence • Emergent evaluation

. . . and finally. . . Everything should be made as simple as possible, but not simpler. Albert Einstein, 1879 – 1955 Reality is complex. Nicoletta Stame, 2004 The art of dealing with the complicated and complex real world lies in knowing when to simplify and when, and how, to complicate. Patricia Rogers, 2008