Monitoring in Outcome Mapping Principles Design Practice Steff

Monitoring in Outcome Mapping Principles, Design & Practice Steff Deprez & Kaia Ambrose OM Lab Dar Es Salaam, Tanzania 22 -23 Sept, 2014 outcomemapping. ca

Monitoring in Outcome Mapping 1. Core principles of monitoring in Outcome Mapping 2. Monitoring Design Issues Experiences from practice 3. Monitoring Practice Approaches, tools & instruments Steff Deprez & Kaia Ambrose (Sept 2014) 2

1. Core principles of monitoring in Outcome Mapping Based on OM Manual outcomemapping. ca

Steff Deprez & Kaia Ambrose (Sept 2014) 4

Outcome Mapping Monitoring • Systematic collection of data on outcomes and performance • A regular learning & improvement cycle • Credit a program for its contribution to bringing about change • Encourages the program to challenge itself Steff Deprez & Kaia Ambrose (Sept 2014) 5

Outcome Mapping Monitoring • • Flexibility Participatory Evaluative thinking Organisational & social learning • Power of self-assessment • Regular face-to-face meetings Steff Deprez & Kaia Ambrose (Sept 2014) 6

Steff Deprez & Kaia Ambrose (Sept 2014) 7

Steff Deprez & Kaia Ambrose (Sept 2014) 8

design boldly M&E modestly within the broadest development context or sphere of interest within the program’s sphere of influence Steff Deprez & Kaia Ambrose (Sept 2014) 9

Outcome Mapping offers a system/process to gather data & encourage reflection on: 1. The progress of external partners towards the achievement of outcomes (progress markers) 2. The internal performance of the program (strategy maps) 3. The program's functioning as an organizational unit (Organisational Practices) Steff Deprez & Kaia Ambrose (Sept 2014) 10

Monitoring in Outcome Mapping Sphere of interest Beneficiary 2 Beneficiary 1 Focus of M&E Beneficiary 3 Outcomes Sphere of influence Boundary partner 1 Boundary partner 2 Boundary partner 3 Sphere of control Behavioral changes Intervention Strategies Efficiency & Relevancy Implementing team Organisational Practices Viability Steff Deprez & Kaia Ambrose (Sept 2014) 11

3 types of monitoring journals Beneficiary 2 Beneficiary 1 Focus of M&E OM Journals Beneficiary 3 Outcomes Boundary partner 1 Boundary partner 2 Boundary partner 3 Outcome Journal Behavioral changes Intervention Strategies Strategy Journal Efficiency & Relevancy Implementing team Organisational Practices Performance Journal viability Steff Deprez & Kaia Ambrose (Sept 2014) 12

Monitoring Plan Steff Deprez & Kaia Ambrose (Sept 2014) 13

Critical questions? Programme response • What should we keep doing? • What do we need to change in order to improve? • Are we still working with the right BPs? • What strategies/practices do we need to add? • What strategies do we need to end? • What should be evaluated in more depth? Steff Deprez & Kaia Ambrose (Sept 2014) 14

2. Monitoring design issues Experiences from practice outcomemapping. ca

Making learning explicit Use of Spaces & Rhythms (*) Conventional M&E design • Information needs related to the programme framework (objectives, results, outcomes, … + indicators) • Data collection methods • Reporting Outcome Mapping (as presented in the OM manual) Based on principles of Utilisation-Focused Evaluation • Focus on monitoring priorities: Who will use it? Purpose? • Use of outcome, strategy and performance journals (*) introduced by Guijt & Ortiz (2008) Steff Deprez & Kaia Ambrose (Sept 2014) 16

Monitoring Plan Steff Deprez & Kaia Ambrose (Sept 2014) 17

Assumptions about monitoring (in OM) • Monitoring process = learning process > reflection and analysis happen automatically • An actor-centered design leads to a participatory monitoring process • M&E results will be used • Users have the capacity, time and willingness to participate or facilitate the monitoring process • Using outcome journals & strategy journals is enough to pave the way forward • The monitoring process is embedded in organisational or programme management cycles • … Steff Deprez & Kaia Ambrose (Sept 2014) 18

Making learning explicit • For learning to happen > data is not the starting point • Start from the intended use • Start with defining the spaces that are crucial for debate, sharing, reflection and decision-making • Make the monitoring integral to the thinking and doing of the organisation and programme Steff Deprez & Kaia Ambrose (Sept 2014) 19

Learning-oriented monitoring (Source: Seeking Surprise (Guijt, 2008) Steff Deprez & Kaia Ambrose (Sept 2014) 20

Three core steps in the design of a learning-oriented monitoring system 1. BE CLEAR ON PURPOSE, USES & USERS 2. DEFINE ORGANISATIONAL SPACES & RHYTHMS 3. DECIDE ON INFORMATION NEEDS Which information is required, for who, at what time/event, in what form, to do what? Steff Deprez & Kaia Ambrose (Sept 2014) 21

1. BE CLEAR ON PURPOSE, USES & USERS E. g. Intended uses of M&E process (Patton) Steff Deprez & Kaia Ambrose (Sept 2014) 22

1. BE CLEAR ON PURPOSE, USES & USERS e. g. Wheel of Learning Purposes (Guijt, 2008) Steff Deprez & Kaia Ambrose (Sept 2014) 23

1. BE CLEAR ON PURPOSE, USES & USERS e. g. PLA system VECO Programme improvement ACCOUNTABILITY Programmatic & financial accountability LEARNING Knowledge creation VECO & partners Negotiation & understanding chain actors Upward & downward accountability Short-term planning PLANNING Evidence building & upscaling Long-term & strategic planning Steff Deprez & Kaia Ambrose (Sept 2014) 24

1. BE CLEAR ON PURPOSE, USES & USERS 2. DEFINE ORGANISATIONAL SPACES & RHYTHMS 3. DECIDE ON INFORMATION NEEDS • What are the spaces and rhythms central to planning, learning, accountability, debate, decision-making, … (Guijt & Ortiz, 2007) • How can we ensure that monitoring is integral to the thinking and doing of the organisation and programme? Steff Deprez & Kaia Ambrose (Sept 2014) 25

2. DEFINE ORGANISATIONAL SPACES & RHYTHMS Organisational spaces Formal and informal meetings and events which bring organisations and programmes to life Rhythms Patterns in time, the regular activities or processes which provide a structure-in-time, through which an organisation can direct, mobilise and regulate its efforts, i. e. regular weekly, monthly, annual activities that characterise the tempo of organisational functioning. When do people interact and share information and make sense of what is happening? Steff Deprez & Kaia Ambrose (Sept 2014) 26

Description of the main spaces & rhythms EVENT What is the purpose of the event Time/ frequency Who participates Who coordinates What is the expected output Which data Information is required … … … > Group exercise Steff Deprez & Kaia Ambrose (Sept 2014) 27

1. BE CLEAR ON PURPOSE, USES & USERS 2. DEFINE ORGANISATIONAL SPACES & RHYTHMS 3. DECIDE ON INFORMATION NEEDS • Which data & information is required? • What type of data / information? • From ‘Nice-to-know’ to ‘Must-know’ information Steff Deprez & Kaia Ambrose (Sept 2014) 28

Steff Deprez & Kaia Ambrose (Sept 2014) 29

EVENT What is the purpose of the event Time/ frequency Who participates Who coordinates What is the expected output Which data Information is required … … … Information needs linked to the main spaces & processes Steff Deprez & Kaia Ambrose (Sept 2014) 30

Masterlist of info needs General info need Specific Frequency information By when? need? (indicators or questions) Who has the info? Or where is the data generated? Data collection Who will use the method/approach info at which ? event?

Plan for Sensemaking 1. BE CLEAR ON PURPOSE, USES & USERS 2. DEFINE ORGANISATIONAL SPACES & RHYTHMS 3. DECIDE ON INFORMATION NEEDS • Plan how data is used analysed > make it usable for action • Focus on social interaction: sharing, reflection, debate, decision • Should be well-planned & facilitated > it will not happen by itself Steff Deprez & Kaia Ambrose (Sept 2014) 32

Institutionalising a learning-oriented M&E practice How to make sure that your monitoring principles and design is translated in an effective monitoring practice? 1. Creating organisational conditions: motives, means & oppportunities 2. The ‘web of institutionalisation’ > Should be reflected in the Organisational Practices Steff Deprez & Kaia Ambrose (Sept 2014) 33

Creating the motives, means & opportunities Creating Motives -Guiding ideas -Support by management -Develop a learning Culture -Provide incentives Creating Means -Human capacities -Specialist support -Concepts, methods and tools -Budget Creating Opportunities -Integration in planning and management -Clear M&E plans and responsabilities -Responsive information management system -Trust and respect – speak out, challenge, feedback (Steff Deprez, 2009) Steff Deprez & Kaia Ambrose (Sept 2014) 34

2. The web of institutionalisation (Levy, 2006) See: THE PROCESS OF INSTITUTIONALISING GENDER IN POLICY AND PLANNING: THE ‘WEB’ OF INSTITUTIONALISATION (Levy, 2006) Steff Deprez & Kaia Ambrose (Sept 2014) 35

3. Monitoring Practice in OM Experiences from practice outcomemapping. ca

Monitoring Practice in OM ü Working with progress markers, boundary partners and organizational practices ü Sense-making with boundary partners ü Ongoing challenges

Thinking through the different aspects of monitoring – M&E plan (Performance Measurement Framework) – more narrative – Unpack different moments Data from grantees • survey • quan and qual Data from POs • trip reports • survey Collate • who? • where? (database) Team • process for doing this interpretation • rubrics / dashboard feed into program level

Working with progress markers • • What is your purpose and use? What is your monitoring culture? What resources do you have for monitoring? What qualitative and quantitative data needs do you have?

Gr. OW program • Show matrix

Improved social and economic status of 35, 000 low-income women in 11 districts in East Harague Intermediate outcome: Increased financial inclusion and entrepreneurship of low-income women Immediate outcome: Improved business and technical skills/capacities of low- income women for enterprise development FSP leaders Immediate outcome: Increased access of lowincome women to financial institutions and services Intermediate outcome: Increased participation of low-income women in household and community level decision making ? Immediate outcome: Increased access of lowincome women to legal and social service providers

Outcome Mapping Logic Model

Challenges • Qualitative data collection – informal interviews, observation (including looking for unexpected – positive and negative) • Qualitative analysis – looking for patterns and trends • Critical analysis and sense-making – the need for facilitated, well-constructed (agenda, exercises) spaces and processes • Usage of information

Evolving Lessons ü Monitoring beyond outputs ü Good Enough (in terms of tools, capacity) and build from there ü M&E – mande – evaluative thinking – - explicit sense-making spaces

Working with progress markers üUse progress markers as a checklist to track progression against pre-defined behavioural changes for a specific partner in a specific period of time; use of scoring (LMH, 1234, colour) üWrite a qualitative description of change (I. e. every 4 -6 months) for each pre-defined PM for a respective period üOther monitoring tools, qual or quan, that are then cross-referenced with pre-defined PMs (new ones added)

Working with progress markers: who? üBP describe their own change - then send to implementing team üImplementing team describes change based on their own observations üMutual reflection process with team and BPs üExternal evaluator judges progression in change

Working with progress markers: what? üEvery single PM monitored üOnly PMs that are relevant for a specific period üPMs and / or OCs used to trigger discussion during reflection process; key changes documented üDepth of analysis can vary üAcross different BPs (comparison) üIn combination with SMs (effective intervention? )

Working with progress markers Using progress markers at an organisational scale (across programmes) üDifferent geographical regions üDifferent thematic foci üDifferent type and bigger numbers of boundary partners

Working with progress markers Use of progress markers for whom? üUse of general & standard progress markers for each type of BP -less (or not) useful for the individual projects and their actors üTailor-made progress markers for individual BPs + relevant to guide and steer local projects and their actors - less useful for higher levels, overload detailed data & aggregation is difficult

Sensemaking with BPs üRegular (4 -6 months) reflection with staff + BP (one or more), external evaluator, other stakeholders üEmbedded in spaces and rhythms of the programme

Working with strategy maps üUseful for monitoring relevance, effectiveness and efficiency of the supporting activities of the implementing team üUnderstand how you spend time and money within a programme team üGet direct feedback from BPs on the required / requested support

e. g. VECO Indonesia - 9 categories of strategy maps Every activity that is carried out by the VECO can be linked to one of the 9 support strategies.

9 categories of strategy maps EXAMPLE MONTHLY OVERVIEW I 1 I 2 I 3 I 4 E 1 E 2 E 5

Working with organisational practices üUse as is in manual üUse of OCs and PMs at the level of the implementing organization üMotives, Means and Opportunities üWeb of institutionalisation

Reports and Journals üSimplified journal formats üCombined PM and SM reports üIntegrate elements from journals into regular programme reports üIntegration in MIS, databases, etc. üUsed as final report, or as report embedded within another report üOM journal as a database üSM as an activity guide

Combined use of OM and the logical framework outcomemapping. ca

Three ways of combining LFA & OM OM+ Modifications of OM framework with elements of LFA+ Modificiations of an LFA with elements of OM Combined uses OM frameworks that are linked to/ feed into LFA

OM+ Additional layer of objectives (+indicators) to pinpoint exactly to what BPs are contributing Some programmes add an ‘ouput’ layer between strategy maps and outcomes

OM+ Additional layer of objectives (+indicators) + intermediate tangible results (+indicators) to clearly state what BPs are contributing

LFA+ In terve n ti on lo gi c In di cat o rs Sou rce of Ver if i cati on A ss um ption s Goal / Devel opm en t O bj Pr oj ect Ob jec tive / Pu rpo se Res u lts / Outc om es A c tivi t ies ≈ Specific objectives + indicators ≈ Outcome Challenge + Progress markers ≈ Activities VECO Logical Framework of a specific project Actor-centred results: results describing changes in behaviour OM Framework of a specific project

Combined use of LFA & OM In terve n ti on lo gi c In di cat o rs Sou rce of Ver if i cati on A ss um ption s Goal / Devel opm en t O bj Pr oj ect Ob jec tive / Pu rpo se Res u lts / Outc om es A c tivi t ies ≈ Specific objectives + indicators ≈ Chain results + indicators Logical framework of the over-all programme Synthesis of the different projects ≈ PMs BPs + Activities VECO OM frameworks are guiding the PM&E of specific projects