Monitoring and Evaluation in Asset Based Approaches Workshop

Monitoring and Evaluation in Asset Based Approaches

Workshop outcomes • • Explain Theory of Change Provide examples of methods and tools Explore ways of providing an evidence base Practical examples of demonstrating impact

Some evaluation terms Outputs Inputs Need Outcomes Activities

Some evaluation terms Need – problem you are solving Input – what goes in Activity – what you do Output – immediate results Outcome – the change

Theory of Change Need Inputs Activities Process Outputs Outcomes Impact

Outputs v Outcomes “We had 100 young people attend our community arts sessions…” “So what? ”

Need Inputs Activity Output

The question……… Has my need been met? ? ?

Two potential outcomes – that we don’t know until we try the cake Outcome – delicious cake Outcome – inedible cake

If you want to understand the difference you are making…. and if you are meeting needs…. . you have to measure the outcomes/changes you are making!

But it is a challenge…. . In groups Draw a person in the middle From your experience Thinks of all of the different things that might influence a persons confidence…

Wider society Community Family and Friends Individual

Tools can be broadly grouped into two categories: • Quantitative tools are generally ones that collect data you can quantify (count!) – they are really useful for telling you what has happened in a project or activity • Qualitative tools are generally ones that produce narrative data – they are really useful for telling you why this happened and how

Types of Tools • The tools are: • Cohort survey – ask the same group of people the same questions at different time points • Individual tracking survey – ask the same person the same questions at different time points • Point in time survey • Observation • Interviews • Stories

Cohort Survey About the Tool • You ask the same group of people the same questions at two different points in time Advantages • It is an easy way to collect information from groups. You can use a variety of tools with the group and it is time efficient to input or analyse. It doesn’t matter if the groups are slightly different at the different time points. Disadvantages • Surveys only show you what has happened not how or why. With a cohort survey you are only tracking the change in the group not individuals

Cohort Survey How to do it Basic • The easiest way to do a cohort survey is to ask the group in one go. This can be done by a paper survey. But it is equally easy to do as a group exercise. For this type of survey you are interested in the progress of the group not individuals. So you can put the questions on flip chart on the wall, use voting buttons or ask people to self select where they stand as a group. Advanced • If you chose questions used in national surveys you can compare your answers to the national picture, You could also use a control group—a similar group of people not included in your project and compare your survey results to this group.

Individual Tracking Survey About the Tool • You ask the same person the same questions at two different time points Advantages • It is a robust way of understanding and tracking change. It allows you to understand how each individual is progressing in the group. You can use workbooks or personal development plans as an easy way of capturing start points and end points. Disadvantages • It can be very time consuming, particularly if working with larger groups. You need a mechanism for tracking people so it means asking people for personal information or a unique identifier.

Individual Tracking Survey How to do it Basic • You need to get responses from people individually. This can be done face to face or via a survey that is distributed to the group. You are likely to need to use some form of paper (or tablet) survey to allow you to do this. You also need a way of tracking individuals to match the surveys up. You are likely to need some form of software (such as Views) to help with this. Advanced • If you chose questions used in national surveys you can compare your answers to the national picture, You could also use a control group—a similar group of people not included in your project and compare your survey results to this group.

Point in Time Survey About the Tool • You ask people once to reflect on the outcomes they have experienced from your project Advantages • It is easy to administer as you only do it at one point in your project. It gives you some indication of the outcomes people feel they have achieved. Disadvantages • People will often over claim the outcomes they have achieved using this method. You will also often miss some of the changes in people depending on when you do the survey. You often need to use a survey to do this or some way of ensuring people can respond privately.

Point in Time Survey How to do it Basic • Use a reflective baseline. Ask people to score on a scale where they felt they were at the start of the project and where they feel they are now. When you input the data you can use the scales to help quantify the distance people report they have travelled. Advanced • If you chose questions used in national surveys you can compare your answers to the national picture, You could also use a control group—a similar group of people not included in your project and compare your survey results to this group.

Observation About the Tool • The person leading or observing the sessions tracks progress of individuals or the group Advantages • It is easy to do and requires little time from participants in the group. It allows the observer to consider changes in individuals and the way the whole group interacts. They work well with groups that struggle with surveys and with participants who may not recognise the changes in themselves Disadvantages • You are making a judgement on the change that has happened in people and so it is your opinion rather than the participants. The observer needs to be systematic so it takes time from the group leader. It can take time to record analyse data.

Observation How to do it Basic • At the end of each session jot down your observations on how the group or individuals have behaved. Do this at the end of every session against a common set of criteria, for example attitude, skills and behaviour of the group. It needs to be done regularly in order for you to be able to track the change. Advanced • There are several published observational tools. These encourage you to consider the progress against a structured set of criteria, normally based on academic research.

Interviews About the Tool • You ask people a set of questions about involvement in the project. Advantages • It is easy to do and gives you key insights into why or how your project is working. They provide lots of information. You can use interesting tools like video or pod casts to collect and then share what participants say. They can be a great way of involving volunteers in the project. Disadvantages • They are generally conducted one to one and so this can be time consuming and take the group leader away from the session. They generate a lot of information that needs to be analysed.

Interviews How to do it Basic • Identify 3 to 5 questions about your project. Use video of audio recordings to speak to a random selection of people attending your session. Once away from the session listen/watch the recordings and identify any common themes from what people are saying. Advanced • Focus groups allow you to collect information from many participants at once. They produce a different set of answers because of the group dynamic but require more skilled facilitation. As with interviews you can use a core set of key questions to prompt the discussion.

Stories About the Tool • People tell you about their experience of being involved in your project or activity Advantages • Stories are brilliant. They allow participants to tell you in their own way what they have got out of a project. You can use writing, drawing, film or audio to collect peoples feedback. You can do it formally or informally at the end of sessions. They can generate information you can use in case studies and to showcase with funders as well as giving you insight into your project. Disadvantages • If you are collecting lots of stories it can generate lots of information for you to analyse.

Stories How to do it Basic • Every couple of months get participants to tell you about the Most Significant Change that has happened to them since being involved in the project. Get to know your group and find the technique that works best for them. Review the stories to identify common themes Advanced • Once you have a selection of stories as a project team discuss which one you think is the best and why. Although this is a purely subjective exercise it helps you to reflect on what matters most to the project and allows you to shape your project to achieve more of it.

What is evidence?

What is evidence? “the available body of facts or information indicating whether a belief or proposition is true or valid” “Assessing the impact of a project, programme or strategy and identifying lessons for the future”

What is evidence anyway? Scientific Method Anecdote/ Insight

Challenges In groups What are the challenges you face collecting an evidence base – particularly but not exclusively around asset based approaches

Some examples Place Based Funding – Sport England Comic Relief Come Outside! – Natural Resources Wales Renew Wales – DTA Wales

Common Features All have subsets of projects trying to achieve different things Tools that allow you to track journeys and different starting points Looking for commonalities and differences Process is really important – understanding the features that led to it being a success

Top Tips 1. 2. 3. 4. 5. Be Honest Use a variety of tools Recognise the value of insight Know your audience Be confident in your findings 6. Get some help!