Service Evaluation Toolkit for Health and Social Care

Service Evaluation Toolkit for Health and Social Care Commissioners Avon Primary Care Research Collaborative 2015

Evaluation Toolkit Introduction How to use this toolkit Improving the quality of health and care is a priority for the NHS. Evaluation an integral part of policy making, strategic planning and commissioning. It can help you make judgements about the policy or service as well as understanding what works and what needs improving. The toolkit is set out in four steps: Assess, Plan, Do and Review. Each provides you with a set of templates and tools to support your evaluation. Used as a whole, this toolkit will help you formulate your evaluation plan, collect and analyse your data, and then use and share your findings. It is likely that you are reading this because you have been tasked with considering how to evaluate a service or intervention or you have been asked to participate in an evaluation. This toolkit has been developed as a practical resource to help you to: • Frame the important questions about your services • Plan a pragmatic evaluation that will answer those questions Share and use the findings from the evaluation • This is a simplified approach, but by going through each section, you will gather the information you need to be able to evaluate your service effectively. The toolkit is part of a series of documents, including a complimentary guide and a plain English summary so that you can evaluate with confidence. It is important to note that while the steps do not necessarily follow a linear process, by completing each step you will have addressed the main points required to carry out a well-planned and successful evaluation. Tools Guided templates Top Tips Resources Note: Service is used through out but relates to projects, programmes, interventions, pilots etc

What is Evaluation? There are multiple definitions for evaluation, many of which have common themes around judging the merit, worth, value or quality of something. Our definition we have adopted locally was produced in collaboration with stakeholders by the West of England Evaluation Strategy Group is as follows: “A study in which research procedures are used in a systematic way to judge the quality or value of a service or intervention, providing evidence that can be used to improve it” (West of England Evaluation Strategy Groups, 2013) Monitoring and Evaluation We routinely monitor the performance of services however monitoring alone can only tell us so much – in that we are doing things – but it unable to tell us if we are doing the right things, what is working and what needs improving. However, monitoring data is a useful source of information to help you to evaluate your service. In turn planning your evaluation alongside your monitoring framework can help to reduce duplication and unnecessary data collection, shape and improve the monitoring data you plan to collect and identify appropriate baseline and benchmarking data. What is Service Evaluation? There are multiple facets to evaluation: • from research to a service evaluation; • from identifying areas for improvement to providing accountability; • from the simple (such as a single survey) to the complex (using multiple methods); and it can be used in many different contexts. A central component to evaluation is one of comparison, either comparing before, with another service or with itself. In health and care evaluation tends to be distinguished as either research or service evaluation. The key distinguishing feature between the two is dependent on the purpose of the study: Research aiming to derive “generalisable” new knowledge beyond the setting it took place; where as A service evaluation is performed to meet a specific local need and is focused on what is (or will form) current care with the evidence collected to inform recommendations not intended to be used beyond the setting it took place (although may be transferable to other areas). This toolkit is focused on helping you to plan and conduct an appropriate evaluation. Health Research Authorities - Defining Research Leaflet - http: //www. nres. nhs. uk/applications/is-your-project-research/ provides accepted definitions for research, service evaluation and clinical audit Charities Evaluation Service http: //www. ces-vol. org. uk/about-performance-improvement/about-monitoring-evaluation guide to monitoring and evaluation

Top Tips These top-tips have been developed from years of experience of supporting those working in the NHS and Public Health with evaluation. They are useful practical tips to support you to start to develop your evaluation plans. Remember evaluations do require adequate resourcing, skills and time so it is important to ensure that your evaluation is feasible, proportionate and appropriate to the service being evaluated and that it will provide you with useful and valuable information. A useful question to ask yourself is “can you afford not to do an evaluation”? 1 E. valua. tion Extremely Valuable Information 2 Ref 1: HM Treasury (2011) The Magenta Book. Guidance for Evaluation Ref 2: Carolyn Cumpsty Fowler, John Hopkins University (2011) http: //www. odh. ohio. gov/~/media/Healthy. Ohio/ASSETS/Files/injury %20 prevention/evaltrainingfowlerpresentation. ashx

Assess Plan Do Review Navigation This toolkit provides you with a suite of resources (tips, tools, templates, signposting) to enable you conduct a service evaluation. You can use the tools and templates as a package or in isolation if you just need support with one aspect. The questions below will help you to navigate through the toolkit or signpost you to specific areas you would like help with. Question Assess • • Do you need to do an evaluation? Do you have an understanding of how your service will deliver the planned outcomes including evidence to support these assumptions? Are you confident you have all the right people involved and engaged? Have you identified appropriate resources to conduct the evaluation? 1 2 Have you developed your evaluation plan including identifying your: • aims and objectives of the evaluation? • evaluation types and approach • data to be collected? baseline data? • ethics and governance implications? 5 6 7 8, 9 and 10 11 Plan • Do • 13. Do you need to develop your data collection tools? • 14. Have you analysed your data? Review • • Toolkit Resource Do you need to write an evaluation report? Who should you be sharing your findings with? Do you need to develop an action plan based on the findings and recommendations? Am I sure I have covered everything - Checklist 3 4 12 13 14 Guide Page No.

Assess Plan Do Review Resource 1. Do I need to be doing an evaluation? Evaluation requires an investment of time, money and resources and so before you begin you need to assess whether you should be conducting an evaluation before making this investment. The following questions will help you to start to consider whether you need to and can do an evaluation, we recommend that you consider these in partnership with your stakeholders. You may not be able to answer them all straight away but useful to keep in mind when planning your evaluation. An Evaluability Assessment may be useful in determining whether the service is ready to be evaluated and what type of evaluation is required. Question Yes Do I know what I want to evaluate and why? Consider drivers and audience Do I already know the answer to my evaluation question? Consider evidence Will I be able to use the results of the evaluation? Consider context Will these evaluation results provide me with useful information that will be considered “value for money”? Do benefits of conducting the evaluation out way costs and consequences. Do I have the resources and/or skills available to undertake this evaluation? Consider how Will the information I collect be able to answer my evaluation question (the evaluations aims and objectives)? Consider quality and accessibility of data Is it the right time to conduct an evaluation of the service? Consider: • Stage of development and complexity of the service • Timescales of the service • Context the service is operating in Are there other options to doing an evaluation? Consider: • Are current project processes such as monitoring enough? • Should I consider other options such as clinical audit, quality improvement or research? For more information on “Evaluability Assessment” the following resources will help: http: //betterevaluation. org/themes/evaluability_assessment No ?

Assess Plan Do Review Resource 2. Understanding your service – Context When considering your evaluation you first need to understand your service in terms of the outcomes you want the service to achieve and for whom (what need are you trying to address? ), what will deliver these changes and in what context. The following are aspects of the service that are useful to consider when planning your evaluation. Aspect Description Implication Purpose What is the purpose of the service? What need is it addressing? What outcomes will it achieve? What is the purpose of the evaluation? How will it be used? Who is your audience? It is important to understand the purpose of your service (i. e. what changes it intends to make – outcomes – and how it intends to do this) and the evaluation (i. e. what answer does it need to address as this will impact on the type of evaluation you need). Evidence base What is the evidence base for the planned service and associated outcomes? What is the strength of evidence? How have similar services previously been evaluated? Understanding the evidence base for your planned service can help inform your evaluation approach and methods: from understanding the strength of the evidence available i. e. if well evidenced then you might focus on the implementation (process) evaluation; to identifying appropriate outcomes and outcome measures; to looking at how others have evaluated similar schemes before. Stage of development Is it new? Has it been in place for a while? Understanding the stage of development of your service will also impact on the type of evaluation you need to conduct. If it is new and being developed you will want to take a more formative (improvement) approach. Level of complexity and context Is it a single change? Are there multiple components? What is the context/environment in which it is working? Understanding how complex your service is and the context in which it is operating is also important. How many services do you know trying to reduce unplanned admissions? Timescales What are the timescales? Is it a one year or five year pilot? Timescales will impact on what you are able to evaluate and how, i. e. if a long term investment you are likely to be able to not only look at implementation (formative/process evaluation) but also outcomes.

Assess Plan Do Review Resource 2: Understanding your service – Theory of change A useful process for planning your evaluation is to develop a “theory of change” for your service. This can be useful way of articulating and providing a visual representation of the links between the various activities of service and how this will lead to the long term outcomes it is trying to achieve. Inputs Activities Outputs Intermediate Outcomes Impact To develop your “theory of change” working with your key stakeholders, some flip chart paper, pens and post-it notes consider the following questions 1: 1. Who the service is for? i. e. your case for change will usually set out the population group, their needs and characteristics, the problem it is trying to address 2. What are the long – term outcomes you want to achieve? Then working backwards 1. What are the intermediate outcomes (short and medium term) that will lead to these long term outcomes? 2. What are the activities that the service will undertake to deliver these intermediate outcomes? 3. What evidence (from research and local learning) is available to inform and support the links between activities and outcomes? i. e. what are your assumptions based on the evidence, expertise and learning? 4. What other factors need to be in place to enable this service to work? i. e. what are your enablers? NPC guide to developing Theory of Change 1 - http: //www. thinknpc. org/publications/theory-of-change/ Kellogg Logic Model Development Guide: http: //www. wkkf. org/resource-directory/resource/2006/02/wk-kelloggfoundation-logic-model-development-guide

Assess Plan Do Review Resource 2: Understanding your service – Accessing and reviewing the evidence base It is important to consider what evidence is available to inform your planning and decision making in terms of the service as well as the evaluation. You will be used to using a broad range of evidence from multiple sources including needs assessments, public health and performance data, evidence from research and best practice as well as expertise and local learning. This evidence can be used to inform your evaluation, help identify the outcomes you hope to achieve and the activities (processes) and outputs that will deliver these outcomes. Understand what evidence is already available to inform not only your service design, but the type and level of evaluation you need. Our evidence toolkit http: //evidence. apcrc. nhs. uk can help you with accessing the best available published evidence from research, evaluation and the grey literature. How does this help me? Understanding your service, the context in which it is operating, providing a visual representation of your service and understanding the evidence and theory behind it can help you to identify what your should focus your evaluation on and the type of evaluation you might undertake. The diagram below tries to outline how this helps.

Assess Plan Do Review Resource 3. Involving and engaging the right stakeholders. It is important to consider who will be interested in this service, evaluation or its findings. Engaging and involving all the right stakeholders right at the beginning is an important step not only in developing your service but also in ensuring a good evaluation. Stakeholders will be any one affected by the service or service change and this will include the users, providers and commissioners of the planned services. Engage all key stakeholders, including the patients and their carers, in the design, delivery and dissemination of the evaluation Who are my stakeholders? You may have already completed a stakeholder analysis as a part of your service planning, however if you have not completed one already then we recommend that you conduct a stakeholder analysis. This is ideally done as a group and could use existing forums such as the project team, steering group or advisory group. Don’t forget to involve your local and organisational experts in patient and public involvement, and your equalities and communications leads to help you. Once you have conducted the stakeholder analysis this can then be used to inform: • Who to involve in the evaluation • What expertise and people are available to support the evaluation (including expertise in data collection, data analysis, communication, patient and public involvement) • Your project, evaluation and communication plans N. B. Make sure you consider the needs of those that are seldom heard and are vulnerable. If you have not already, consider completing an equalities impact assessment for the planned service or service change. The NHS Institutes tool, part of its quality improvement tools, http: //www. institute. nhs. uk/quality_and_service_improvement_tools/stak eholder_analysis. html will help you to conduct a stakeholder analysis. For more information about involving patients and the public then INVOLVE www. invo. org. uk, who are funded by the National Institute for Health Research (NIHR) to support public involvement in NHS, public health and social care research, are a useful resource.

Assess Plan Do Review Resource 4. Resources It is important to consider what resources are required for the evaluation from the outset. This may need to be reviewed throughout the planning process. The resource requirements will be informed by the type of evaluation you need (and quite often impact on the design of your evaluation). This is an important step and should be considered early when planning and developing your service (i. e. business case, QIPP scheme, service specification, intervention, programme or project). Plan your evaluation early in the commissioning cycle. This will enable you to ensure that you use the most appropriate methodology, collect the right data, allocate adequate resources and set appropriate timescales. Question Who is responsible for overseeing the delivery of the Evaluation? Who should be involved in the evaluation? Who can help you to deliver this evaluation? Evaluation Resources There will usually be an identified lead for the evaluation who can often be the project/service manager or the commissioner. You will also want to consider your reporting (governance) arrangements for the evaluation, this may be an established project group or committee. Who are your key stakeholders? Are there local experts that can help? Discuss lay involvement with your PPI lead - service users and their families are often well placed to help evaluate the services. What resources are available to support the This could be money and/or staff time depending on the evaluation. If you are conducting an evaluation? Include financial resources internal evaluation then you will need input from staff. If you need to commission an external evaluation then you will need to also identify a budget (approximately 10% of the project value). Utilise existing data and project resources where possible and appropriate. Who will conduct the evaluation? Evaluations can be conducted internally, externally by an independent organisation or a mixture of the two. What you decide will depend on the purpose of your evaluation (i. e. if it is to inform and improve then an internal evaluation may be adequate; if it is to demonstrate a level of accountability then you need to consider an external evaluation), the level of risk involved to the organisation and the participants and the skills and resources available. Our service evaluation guide and website tbc provides you with guidance on commissioning an independent external Evaluation and template to set out your invitation to tender.

Assess Plan Do Review Resource 5. Evaluation Planning Template This template sets out the key elements to consider when planning your evaluation. Don’t forget to build the evaluation into your project plans and processes as much as possible. Make sure it is feasible and utilises appropriate existing data that is aligned to, and informs, your monitoring plans. Title Project team Something that explains what you will be doing. Outline who makes up the team and what their roles will be. Background Outline the context or rationale for undertaking the evaluation. Bring in here information from your business case, service specifications and evidence reviews i. e. what is your case for change, what outcomes do you hope the service will achieve. Ensure that your evaluation has a clear purpose and focus. To do this set your aims and objectives at the beginning with your key stakeholders. It is important to get them right at the outset as this will guide your evaluation design. The overall aim will be a statement of what the evaluation will do. The objectives are narrower statements of what the evaluation will try to achieve in order to meet the overall aim. Aim and objectives Resource 6 Design and methods Resource 7 - 10 Ethics and Governance There are multiple approaches to evaluation which can be complex and confusing. A simple approach to overcome this is to consider the types of information and data you need to answer your evaluations aims and objectives. Consider both qualitative (i. e. narrative data from interviews, focus groups) and quantitative (i. e. numerical data from surveys, monitoring forms), the sources of that data (does it already exist or do you need to collect it). Ensure that you baseline your information where possible and consider utilising benchmarking data. Ensure you have considered what the ethical implications of the evaluation are and how they will be mitigated and reviewed throughout the project lifecycle. Resource 9 Resources Resource 4 Outputs and impact References Outline the resource requirements and include here internal project resources to support the evaluation as well as any ring-fenced financial resources (and funding available). You have already started to identify these when assessing whether to do an evaluation. This is a description of what will be produced (e. g. the evaluation report) from the evaluation as well as the intended use and impact. Remember to build this into the services communication plans. All sources quoted in the proposal should be acknowledged and correctly referenced. Health Foundations Evaluation: What to Consider Guide http: //www. health. org. uk/publication/evaluation-whatconsider is a very useful overview of what to consider when planning an evaluation

Assess Plan Do Review Resource 6: Setting Aims and Objectives It is important to get your aims and objectives right at the outset as this will guide your evaluation design. Involving key stakeholders in setting the aims and objectives of the evaluation helps elicit buy-in; promotes cooperation; ensures all interests are accounted for; and that expectations are managed. The overall aim will be a statement of what the evaluation will do. The objectives are narrower statements of what the evaluation will try to achieve in order to meet the overall aim. The case study below summarises a real-life process of co-producing evaluation objectives with a stakeholder group conducting a training pilot in partnership with volunteers to increase the skills and knowledge of people with Fibromyalgia. The group met to discuss what was important for each of them and considered how the objectives linked together. Note how some of the objectives meet the needs of more than one stakeholder. Stakeholder What they want from the evaluation Agreed Objective The Lay Representative To see how the project is improving experience of living with the condition in order to influence NICE guideline review The Commissioner Is the intervention cost-effective? To evaluate the impact of the training on patient knowledge and skills for enhanced self-care of fibromyalgia To measure stakeholder satisfaction with the approach To understand the potential of the project to effect cost-savings through a reduction in unnecessary hospital admissions and unnecessary GP attendances. The Service Provider Are the aims of the project met? (ie in this case, do patients’ knowledge and skills to self-manage increase? ) The Voluntary Sector Partner Did volunteers help meet the aims of the project? What was the experience of Volunteers in the project? Do volunteers have the potential to deliver cost-savings? Does it fit in with the CCG’s strategic priorities? To evaluate the impact of the training on patient knowledge and skills for enhanced self-care of fibromyalgia. To measure stakeholder satisfaction with the approach What do service-users think of the project? To evaluate the impact of the training on patient knowledge and skills for enhanced self-care of fibromyalgia. To measure stakeholder satisfaction with the approach To understand the potential of the project to effect cost-savings through a reduction in unnecessary hospital admissions and unnecessary GP attendances. AIM: to assess the impact of the pilot to make recommendations to commissioners about future funding

Assess Plan Do Review Resource 7: Types of Evaluation – which type of approach should I use? There a number of different types of evaluation, the most widely used are formative, Process and summative (outcome) evaluation. The following information provides you with a guide to the types pf evaluation, when they are used and the types of approaches and data collection methods that are associated with them. (Roberts, Cavill and Rutter 2009) Process Formative Summative • Assesses implementation and delivery of the initiative, and whether this was carried out as planned • Takes place throughout the project implementation and delivery phase. • Demonstrates to stakeholders that project Objectives are being met • Qualitative methods: Interviews; Focus Groups; Questionnaires • Quantitative methods: Metrics, Benchmarking, quantifiable Performance, Financial, Service Usage data • Service Improvement methods: PDSA Cycles; Rapid Evaluation Cycles • A type of process evaluation that assesses how the intervention / programme can be improved as it is being implemented • Takes place before the project starts and throughout pilot phase to inform implementation. • Shows what worked well and why, and identify challenges to capture lessons learned (continually improving) • Utilises evaluation theory e. g. Theory-of-Change; Realist Evaluation, Experience-Based Design, Critical Path etc. • Helps to assess needs; clarify theory-of-change; to identify areas for shared learning and improvement • Qualitative methods: Literature Review; Interviews; Focus Groups; Observation; Diaries; Questionnaires • Service Improvement methods: Stakeholder Analysis; Logic Modelling; PDSA Cycles; Rapid Evaluation Cycles • Was the project successful? Has the met its predefined aims and objectives? • Looks at the Outcomes (i. e. measures the changes that have occurred as a result of the programme) and/or Impact (i. e. the longer, deeper, systemic changes). • Demonstrates to stakeholders if project aims were met. • Qualitative methods: Interviews; Focus Groups; Questionnaires • Quantitative methods: Metrics, Benchmarking, quantifiable Performance, Financial, Service Usage data • Can be Experimental (e. g. randomised controlled trial), Quasi-Experimental e. g. comparing before and after or use service Improvement methods: PDSA Cycles; Rapid Evaluation Cycles http: //www. ihi. org/resources/Pages/Howto. Improve/default. aspx http: //www. civicpartnerships. org/#!quantitative--qualitative-eval-methods/c 1 bel http: //www. institute. nhs. uk/research_and_evaluation/general/useful_resources. html

Assess Plan Do Review Resource 8. Data Collection Planning Template During the initial planning phase you would have identified the purpose of your evaluation (aims and objectives), the type of evaluation you might need as well as starting to consider the types of data and information you need to be able to answer your aims and objectives. Use the information you have identified so far as well as resources 9 and 10 to complete the following table, setting out your objectives against the data you plan to collect, the sources of that data and who is responsible for collecting and analysing the data. Identify and utilise existing data and data collection methods, where it is available and of good quality, to avoid duplication and unnecessary data collection. Make sure you collect baseline data, consider benchmarking and utilise validated tools where they exist. Consider using multiple (audit, quality improvement) and mixed (qualitative and quantitative) methods. Evaluation Objectives Evaluation questions What are your What are the questions you need to answer to evaluation objectives? enable you to demonstrate whether you have achieved your objectives? Data Sources / Tools Responsibility and Timescale What data do you need to be able to answer your specific questions / measures? Where can you source that data from? Or What data collection tools do you need? What sample size will you need? What baseline and benchmarking data is available? Who will be doing what and when? Objective 1 Objective 2 Explore your evidence review to see how others have evaluated similar initiatives including whethere any validated tools (for example surveys, patient reported outcome measures, patient reported experience measures) available that you can use?

Assess Plan Do Review Resource 9. Data Collection - Decision Tree Tool To help you decide how to collect the information or data you need to support your evaluation the following decision tree can help. Think about the types of information you need and then use this as the starting point to navigate the tree to identify a suggested method for collecting the information. This decision tree was developed in 2005 by Marsh and Glendenning so remember that with advances in technology there also other ways to collect the information such as online surveys, text messaging, video messaging, social media, apps and other mobile technologies.

Assess Plan Do Review Resource 10: Types of data and information Having an adequate baseline for your key metrics is extremely important as a key aspect of evaluation is comparison, whether it is comparing with it self, before or after the service change or with another service. Benchmarking is also a useful tool to compare with other similar services or with national trends. QUALITATIVE • Uses narrative or descriptive data rather than numbers. For example, a description of the views and attitudes of those receiving/providing a service. QUANTITATIVE • Gives numerical results. For example, the percentage of participants completing a programme. BASELINE DATA BENCHMARKING DATA • This is data for the period before the service change, to compare with the period after implementation of the change. • This is performance data on your service that can be compared with similar data collected on other services locally and nationally. PRIMARY SECONDARY • This is additional new data collected for the specific purpose of the evaluation. • This is existing data and will include information you or others already collect. For example, monitoring data, local and national audits, census data. SAMPLING • It is important to consider sample size, i. e. how many people from the population you need for your evaluation. It is important that the sample size is feasible, practical and reasonable/proportionate for the evaluation planned. • There are different approaches to identifying your sample. These include ‘convenience sampling, purposive sampling, snowballing sampling.

Assess Plan Do Review Resource 11. Am I doing an evaluation? Once you have identified the need to do an evaluation and planned your approach it is important that you check that you are doing a service evaluation and not research. To do this we recommend you use the Health Research Authorities tool to help you to assess this or contact your local experts such as your research and development department. http: //www. hra-decisiontools. org. uk/research/ We suggest that you keep a record of the result as a part of your evaluation plan as this acts as an audit trail. We also recommend that you check with your own organisation to ensure that you comply with local arrangements when it comes to the governance and ethics of your evaluation. Ethics and Governance If you are doing a service evaluation then you will need to comply with your local and/or organisational ethics and governance processes. These local and organisational processes may vary and so it is important to ensure that you obtain the right advice and seek the right permissions. If you are unsure check with your local experts, these could be a Research and Development lead, Clinical Governance lead, an Evaluation lead or a University Ethics department. Engage with your local research and information governance experts As a service evaluation you do not need any formal ethical approval from an NHS Ethics Committee, howevery service evaluation should ensure they apply best practice when it comes to ethics ensuring that adequate safeguards are put in place and that the benefits of conducting an evaluation outweigh the risks. This will involve reviewing your evaluation during the design, delivery and dissemination to ensure any ethical issues are identified and actions are put in place to address them. This will include ensuring appropriate informed consent is obtained from participants, confidentiality and anonymity are maintained and the study complies with your organisation‘s information governance and data protection policies and procedures. Particular consideration needs to made for any participants that may be considered vulnerable such as children and people who lack capacity.

Assess Plan Do Review 12. Reporting Template The template below outlines the key information you need to provide within your evaluation report. Note: use this as a guide as your own organisation may already have a reporting template which incorporates your organisations house style. Heading What to include Executive summary Provide a high level summary of your evaluation on a page (max 2 pages) Introduction Outline the purpose of the evaluation, project/evaluation team (who is involved in the evaluation), structure of the report Background Provide the National and Local context for the service and evaluation (why it is important, case for change, evidence base) including information from your business case and evidence review Aims and objectives Clearly set out the aims and objectives of the evaluation and how they were developed / identified Methods Using the information in your evaluation plan set out the overall approach to the evaluation, specific data collection methods/sources (include who, where, when, how and how many) and approach to data analysis. Note here any limitations of the methods. Results Present the key results and analysis from the data collection here (you may decide to include some results in the Appendix if you have lots of data). Findings Present an interpretation of your findings here, bring together different data that work together (triangulation). Include here any limitations of the data and findings and be careful with the language you use (i. e. you are unlikely to be able to show cause and effect) Discussion Relate how the key findings fit with the local and national context, what the key learning is from the evaluation. Conclusion and Recommendations Provide evidence based conclusions to make evidence informed recommendations from the evaluation Appendices Include here your outputs from your worksheets plus any additional data or information to help the reader understand your evaluation. Charity Evaluation Service guide to report writing http: //www. ces-vol. org. uk/Resources/Charities. Evaluation. Services/Documents/Writing%20 better%20 evaluation%20 reports%20 Final. pdf

Assess Plan Do Review 13. Action Planning template Once you have reviewed the results of your evaluation (and taken account of limitations of these findings) you can use these to make your recommendations for going forward based on the evidence. Ideally these would be reviewed by and agreed with your project team and tested out with your key stakeholders. The next step is to develop an action plan for implementing these recommendations. Your organisation may already have a standard template for this, however the following is a simple tool. To maximise the impact of your evaluation make sure you use your findings to inform your decision making going forward And identify areas for improvement Recommendation Action Why Who When Risk level / Issues Recommendation from the evaluation specific action to help address the recommendation explanation of why the action will lead to the desired change Lead responsible for delivery Timescale Use your organisations risk matrix Note any risks or issues associated with this activity NHS Scotlands Quality Improvement Hub Guide to Action planning http: //www. qihub. scot. nhs. uk/knowledge-centre/quality-improvement-tools/action-plan. aspx

Assess Plan Do Review 14. Share your findings (Dissemination/Communication) planning template Working with your organisations communications experts and project team draw up a dissemination plan for your evaluation to maximise its impact. Consider who your audience is, what the different approaches are to communicating the messages and the different formats these could be communicated in and by. Where available, build into your project/service communications strategy. Use multiple formats to communicate your findings for the greatest impact and where possible utilise existing forms and channels of communication (engage your communications experts!) Audience Approach Format Who When Risk level / Issues who you are trying to reach e. g. Participants in the study how you are going to reach your audience) e. g. Website, Staff meeting, newsletter what your message will look like) e. g. Report, leaflet, poster etc Lead responsible for delivery Timescale Use your organisations risk matrix Note any risks or issues associated with this activity Better Evaluation blog on infographics http: //betterevaluation. org/blog/infographics_to_make_your_eval_results_go_viral NHS Wirral’s guide to dissemination http: //info. wirral. nhs. uk/document_uploads/evidence-factsheets/19 disseminatethefindings 11052011. pdf

Evaluation Planning Checklist (1 of 2) Front Planning Checklist Planning is a crucial part of the evaluation process. The following checklist (based on the original APCRC checklist) has been designed to help with the process of carefully planning your evaluation, outlining the key steps going forward. Identify Study type Project Governance Project resources Support for the project: All Stakeholders Questions to ask 1. Is this a service evaluation or research? 2. Who will take responsibility for getting the evaluation off the ground and oversee its delivery? 3. Who will write the evaluation plan, identify the required resources and engage stakeholders? 4. Do you need an advisory or steering group? 4. What resources are available to support this evaluation? What resources do you think you might need / how much might it cost? Who might fund the evaluation? 5. What level or type of evaluation do you need? 9. Who are your key stakeholders? 9. Who needs to be informed? 10. Who do you need to involve in the evaluation planning, delivery and dissemination? 10. Who has skills, experience and expertise to support you with your project? i. e. Patient and public involvement, equalities, communications and engagement, evaluation leads in your own or partner organisations Support for the project: Service User Involvement 14. How will you involve service users, patients, carers and the public in your evaluation? Consider this in terms of the design, delivery (data collection) and dissemination (communicating your findings). Context: Evidence Base 19. 20. What is the evidence base for the planned service, service change, pilot? How have similar services been evaluated in the past? Context: Understanding the Service 19. 20. Is it clear who the service is for? (population group, needs and characteristics) Is it clear what the desired intermediate and long term outcomes are and how the activities of the service or intervention will lead to these? Scope of the project 19. 20. 21. Have you agreed with your stakeholders the purpose of the evaluation? Are you clear what the evaluation will focus on? Is it clear why you are conducting an evaluation?

Evaluation Planning Checklist (2 of 2) Back Identify Questions to ask Aims and Objectives of the 23. Have you engaged your stakeholders to help you identify your evaluations aims (why you are doing evaluation this evaluation) and objectives (what you are trying to achieve)? 24. Are your aims and objectives SMART? Evaluation approach 25. What evaluation approach or method are you planning to take? 26. Do you need to commission an external evaluation? Data requirements 26. What information and data do you already have available to support your evaluation? 27. What additional data collection do you need to undertake to be able to answer the aims and objectives of your evaluation? Data Collection, Analysis 26. What data do you need to collect? Will your data collection tools work? Are there any validated and Reporting tools that can help? Who will collect the data? 27. How will you analyse the data? Who will analyse the data? 28. How and who will write up the findings? 29. Have you identified any training needs to support these activities? Step 5: Evaluation Plan Timescales, responsibilities 28. What are the timescales for the evaluation and the data collection? and resources 29. Who will be responsible for each of these? 30. Do you need any additional resources or funding? Information Governance 30. Have you reviewed your organisational policies on Information Governance including data protection, storage and use to ensure that your evaluation plan complies or sort advice from your information governance lead? 31. Have you sought relevant permissions to undertake the service evaluation? Ethical implications 32. Have you considered the impact of your evaluation on the participants and the service? 33. Have you put adequate safeguards in place to protect the participants in your study? Including gaining consent and feeding back findings. Recommendations and 34. Have you agreed your recommendations and how you are going to implement them? action planning 35. Have you developed your action plan? Sharing the findings Are you still doing an evaluation? 35. How are you going to feedback the findings from your evaluation to your stakeholders? 36. Have you developed a communication plan to share your findings and recommendations? 29. Once you have planned your evaluation recheck whether you are doing a service evaluation or research to ensure that you have the appropriate permissions and approvals for starting the project.

Additional Resources There a number of resources available to support your evaluation, some are included within the toolkit to support specific aspects of your evaluation, this section provides you with other toolkits that may be of use to you. Evaluation Toolkits • CLAHRC Evaluation Guide: Developed by NIHR CLAHRC Leicestershire, Northamptonshire and Rutland in 2012 for clinicians and NHS Managers to help guide them through the process of evaluation – • http: //www. clahrc-cp. nihr. ac. uk/wp-content/uploads/2012/07/Evaluation_GUIDE. pdf NHS Cambridge Full Evaluation Toolkit: Adapted from the PRIMARY CARE SERVICE EVALUATION TOOLKIT Version 1. 5 Peter Marsh and Robert Glendenning to support NHS Cambridge CCG with Evaluation – http: //clahrc-cp. nihr. ac. uk/wp-content/uploads/2012/07/Full_Evaluation_Toolkit. pdf Evaluation Guides • Magenta Book: Developed by HM Treasury to support evaluation of policy – • MRC Framework: Developed by the MRC to help with evaluating complex health interventions – • http: //www. hm-treasury. gov. uk/d/magenta_book_combined. pdf http: //www. mrc. ac. uk/Utilities/Documentrecord/index. htm? d=MRC 004871 Health Foundations Evaluation: What to Consider Guide provides insight into the things you need to consider when planning an evaluation, the different types of evaluation and different methodological approaches – http: //www. health. org. uk/publication/evaluation-what-consider Other useful resources • Charities Evaluation Service Evaluation Tools and Resources – • NPC Evaluation Tools and Resources – • http: //www. ces-vol. org. uk/tools-and-resources http: //www. thinknpc. org/ Social Value UK (SROI network) guide to Social Return on Investment – http: //socialvalueuk. org/what-is-sroi/the-sroi-guide