Eyes Wide Open Mechanisms that trigger use influence

Eyes Wide Open Mechanisms that trigger use / influence AES Conference, Sydney 2011 Heather Nunns, Mathea Roorda & Vicki Wilde

Our aim • is to be more intentional about evaluation use / influence: – to start an evaluation with our ‘eyes wide open’ to opportunities to encourage, promote evaluation use & influence

What are mechanisms? • Mechanisms are the underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest. They are: – usually hidden – sensitive to variations in context – generate outcomes (Leeuw, 2010)

Why focus on mechanisms? • Identifying mechanisms that link cause and effect relations is “crucial for the development of deeper and more fine-grained explanations of social phenomena” (Leeuw, 2010)

Mechanisms of use & influence (Mark & Henry 2004) Level of analysis Type of process / outcome Individual General influence Elaboration Heuristics Priming Skill acquisition Interpersonal Collective Justification Persuasion Change agent Minority-opinion influence Ritualism Legislative hearings Coalition formation Drafting legislation Standard setting Cognitive & affective Salience Local descriptive norms Agenda setting Policy-oriented learning Motivational Personal goals and aspirations Injunctive norms Social reward Exchange Structural incentives Market forces Behavioural New skill performance Individual change in practice Collaborative change in practice Program continuation, cessation or change Policy change Diffusion Opinion and/or attitude valence

Our mechanisms of use / influence Mechanism Alignment Engagement Pertinence Explanation How it can work Alignment of Credible evaluators epistemologies, values Credible methods (personal, Fit with context(s) organisational) Involvement of commissioner and stakeholders Spiral effect creates ownership, understanding, colearning Relevance and timeliness of evaluative findings Responsive to context Translation Accessible & digestible Type of process / outcome Cognitive & affective Motivational Behavioural General influence

Triggering a mechanism • ‘Helping’ conditions, e. g. – closeness to decision makers – level and nature of stakeholders’ engagement – serendipity

Triggering a mechanism • ‘Impeding’ conditions, e. g. – dysfunctional environment – apathy or resistance to evaluation – wrong time / wrong people / wrong place

So what does this mean for evaluation practice? • It’s important to understand mechanisms of use/influence because they are: • what create (or hinder) evaluation use/influence • where we need to direct our attention and efforts

Puzzles and questions • Are any key mechanisms missing? • To what extent are our mechanisms New Zealand-specific? • How to influence mechanisms at the interpersonal and collective levels ie: those beyond our ‘line of sight’? • How do we collect ‘evidence’ of mechanisms at work at these level?

References • Astbury, B. and Leeuw, F. (2010). Unpacking Black Boxes: Mechanisms and Theory Building in Evaluation. American Journal of Evaluation, Vol. 31: 363. Downloaded from http: //aje. sagepub. com/content/31/3/363 on 20 February 2011 • Henry, G. and Mark, M. (2003). Beyond Use: Understanding Evaluation’s Influence on Attitudes and Actions. American Journal of Evaluation, Vol. 24: 293. Downloaded from http: //aje. sagepub. com/content/25/3/293 on 20 February 2011. • Mark, M. , and Henry, G. (2004). The Mechanisms and Outcomes of Evaluation Influence. Evaluation, Vol. 10: 35. Downloaded from http: //evi. sagepub. com/content/10/1/35 on 15 February 2011.