Welcome We will start soon Adapting your monitoring

Welcome! We will start soon.

Adapting your monitoring & evaluation

About the team

Before we begin. . . Please mute yourself during the presentation. You can unmute during the break-out group discussion, or use the chat box to contribute. Use Chat for questions. Clarifying questions will be answered as we go. Other questions will be answered at the end or afterwards via email.

About today’s webinar 20 minute presentation on 3 key questions to ask ourselves at the moment 20 minute breakout group discussion to share with and learn from others 10 minutes to feedback from group discussion

3 key questions to ask ourselves

1. What information do we need to make decisions? Identify the most important questions that you need to have answers to. Which of these are urgent now? Which of these aren’t urgent right now but will become important in the medium-term?

Example You work for a local community centre that runs a food bank and an advice service to support people with benefits, housing and/or employment issues. The food bank is still running and the advice service is now provided via phone only.

Example Your key questions might be: ● ● ● What are the key issues that people need help with when they ring our advice service? Do some groups have different issues to others? (So we can tailor our support) How many people are using our advice service and food bank? How does that compare to before COVID-19? (So we can monitor demand allocate resources). How many of those people are first-time food bank users? (So we can understand changing needs) Who are we not reaching? (So we can identify people who might be falling through the gaps) What do people think about our advice service and food bank? What difference is it making? (So we can understand what good looks like, maintain good quality, and spot ways to improve it). To what extent are we improving people’s wellbeing? (So we can understand the long-term effect of the service and show its importance).

Example Then they might prioritise them. . . ● ● ● What are the key issues that people need help with when they ring our advice service? Do some groups have different issues to others? How many people are using our advice service and food bank? How does that compare to before COVID-19? How many of those people are first-time food bank users? Who are we not reaching? Are there any people who we know were regular food bank or advice service users who we haven’t heard from since the crisis began? What do people think about our advice service and food bank? What difference is it making? To what extent are we improving people’s wellbeing?

More info: https: //www. inspiringimpact. org/learn-to-measure/plan/decide-what-data-to-collect/ 5 types of data User data: Information on the characteristics of the people you are reaching. Engagement data: Information on how service users are using your service, and the extent to which they use it. Feedback data: Information on what people think about the service. Outcomes data: Information on the short term changes, benefits or assets people have got from the service. Impact data: Information on the long-term difference that has resulted from the service.

2. Is it possible to collect that data? Is it safe and ethical? ● ● ● Can we follow government guidelines about social distancing while collecting the data? Is it an appropriate question to be asking someone at the moment? Can we collect data in an accessible and inclusive way so that everyone can participate if they want to? Is it logistically possible? ● ● ● Do we have staff/volunteer capacity to gather, store and analyse that data? Do people have the knowledge, skills and technology required to provide data via phone or online if necessary? Is there relevant existing data from our organisation or others that we could use?

Example What are the key issues that people need help with when they ring our advice service? Could pull topics from call log and tally up common issues, then ask staff to record this more consistently in the future. Could compare this with other community centres in nearby boroughs if they’re happy to share. Who are we not reaching? Not sure how accurate our demographic records are, so let’s look into this! We could also identify regular food bank or advice service users who we haven’t heard from since the crisis began. What do people think about our advice service & food bank? What difference is it making? At the end of advice service calls, we could ask people if they’d be happy to be contacted again in a month to ask them a couple of

Keeping people and their data safe The ICO encourages us to keep sharing data, but. . . ● ● Keep it clear - be clear, open and honest with people about what you are doing with their personal information. Keep it lawful - do you have the person’s consent? Or would they expect you to collect/use/store their information in that way? Keep it secure - share files securely online, use password protection Keep it to a minimum - only collect, use and keep what you actually need! From https: //ico. org. uk/about-the-ico/news-and-events/blog-community-groups-and-covid-19/

3. What am I going to do with the data? Who needs access to the data? What do they need to be able to do with it? How can I make sure that my colleagues get the data they need?

Example What are the key issues that people need help with when they ring the advice service? Could pull topics from call log and tally up common issues, then ask staff to record this more consistently in the future. Advice service staff need to know FAQs so they can be better prepared to answer the questions. I could send staff a weekly email briefing with top issues from the previous week and a reminder of the key advice we’re giving. Our comms officer needs to know the top 3 issues to support fundraising drives and to include in press releases. I could send them a fortnightly email with the top 3 issues and an anonymous example of each.

3 key questions 1. What information do we need to make decisions? 1. Is it possible to collect that data? 1. What am I going to do with the data?

https: //www. inspiringimpact. org/learn-to-measure/

Let’s hear from each other In your break-out groups: ● ● What are your key challenges at the moment? What advice or support do you need? How have you already adapted your monitoring and evaluation? What has worked and what hasn’t worked so well?

https: //www. inspiringimpact. org/impact-support-during-covid-19/

https: //www. inspiringimpact. org/impact-support-during-covid-19/

Sign up to our newsletter to stay up to date. Don’t hesitate to get in touch to let us know what would be most helpful for you right now. www. inspiringimpact. org

Breakout group notes

Group 1 - Amira What are your key challenges at the moment? What advice or support do you need? ● How do you change monitoring and evaluation when you change your delivery model? ○ Funders have been accepting that M&E will be different. ● Changes in user need, developing new products to move to online delivery. ● What kind of data is ethical to collect? ● How do we collect data to support other teams ● Evaluating an online event ● Things are quite fast-moving - not enough time to think and plan. Moving to a more iterative M&E approach - this can also mean different types of data (focusing on what’s good enough right now - test and learn approach) ● There’s a lot of ‘noise’ - lots of advice, people talking about the same issues ● How do you measure things that allow us to make decisions about sustainability? ● Dilemma b/w furloughing staff and continuing to deliver - what data would help make that decision? ● Advice and support: ○ Good to know what other people are doing - particularly around ethics ○ How to simplify things - what are small things we can do to get the data we need How have you already adapted your monitoring and evaluation? What has worked and what hasn’t worked so well? ● Moving towards more quantitative approaches - rapid response, getting in key data quickly -previously highly qualitative ● Indirect data gathering - from staff who have relationships with service users. Might not be best practice but works in the short term ● Charities have been calling every service user to get a sense of how they are and what their situation is

Group 2 - Antonia Challenges: Small charity with small pot of funding with 1 staff… but not done any M&E work yet so looking for new ideas on where to start. Hard to collect information from young people who don’t want to engage. There assumptions that people have access to online provision. How to answer evaluation questions in funding bids? How do you evaluate an emergency response when you are focusing on immediate need. Adapting / opportunities Conversation with funder about realistic expectations. E. g could more in-depth evaluation to happen at a later stage? Focus on quick, verbal feedback for now. One org circulated a survey which was v really helpful to see what support is needed as very different to before Covid 19. Can we turn to international development charities for their learning, tools, frameworks? E. g. UK Evaluation society; https: //blogs. ncvo. org. uk/2020/04/09/five-lessons-from-the-past-about-spontaneous-volunteering-in-emergencies/

Group 3 - Ben What are your key challenges at the moment? What advice or support do you need? ● First step: health of organisation. Hard to focus on evaluation when fire fighting ● Some services impossible to deliverat the moment / taking time to transfer to online ● Speed of change is challenging. Prioritisation and data collection needsare ever changing ● Using evaluation data to think about the futureis hard with so many unknowns ● People working from home - things take longer How have you already adapted your monitoring and evaluation? What has worked and what hasn’t worked so well? ● Questionnaire to supporters/membersasking what they want at this time ● Shifting to online has increased participation ● Limited resources: falling back to what workedprior to the crisis ● Everyone stretched and overloaded so need to communicate more personally ● Being more flexible with what countsas data/learning

Group 4 - Catherine What are your key challenges at the moment? What advice or support do you need? How have you already adapted your monitoring and evaluation? What has worked and what hasn’t worked so well? Wide range of orgs in the group ● ● ● Lots of services moving online – and so feedback gathering moving online. Implications of this for those that are digitally excluded. some just not able to use services. Some changes to data gathering to inform services now (i. e digital access / inclusion). Cascading info up to council / funders etc. Can use info in the future for advocacy Some implications for funding distribution raised Some points around services that have been moved online that are taking a much different amount of time /resource to deliver (ie. Group session of 2 hours for 6 -8 people , now taking all day for same amount of interaction in 1 -1 calls) Are we collecting same as historical info ? or do we need new? What next – longer term implications Dimensions of covid crisis. What does this look like longer term? Months and years to follow in terms of poverty and mental health etc

Group 5 - James What are your key challenges at the moment? What advice or support do you need? How have you already adapted your monitoring and evaluation? What has worked and what hasn’t worked so well? Impact Measurement not top of mind - but there are some important questions. - Most of us working through partners - We’re delivering services very differently - so need to understand how that is affecting things. Being agile. Need to think / react quickly. ‘Designing while delivering’. How do we compare this to what has happened before? Do we achieve more / less? Where is the need right now? How do we find out? - Making use of the data we have got - compromises. - What are the good ideas right now? Phone Contact advice from Evaluation Support Scotland

Group 6 - Mair What are your key challenges at the moment? What advice or support do you need? How have you already adapted your monitoring and evaluation? What has worked and what hasn’t worked so well?

Group 7 - Nicola What are your key challenges at the moment? What advice or support do you need? ● ● ● Consultant - having to adapt our products very rapidly. How can we keep evaluation on organisations’ radars? It doesn’t feel like a priority at the moment Things are shifting and changing so rapidly for people at the moment Working with the same group of women but we are more busy than usual, their needs are more intense. The intensity of our work has increased The way we deliver our activity has changed from face to digital. Now producing videos for people in the community. Concerned about our reach. Who is falling through the gaps (ie are vulnerable children not attending school falling through the net. How would we know? ) We are delivering an emergency service in partnership with other organisations but we are all using different data collection systems. What does this mean for our impact? How have you already adapted your monitoring and evaluation? What has worked and what hasn’t worked so well? ● ● ● Agreed the 5 types of data is really helpful. Great starting point and a way of prioritising what we should be collecting. What do I need to know? Agreed that responsive/swift solutions are helpful, welcomed the II webpages as a resource Be more flexible around what counts as evidence - look for and use more anecdotal feedback