Rotherham Building Stronger Communities Draft Theory of Change

Rotherham Building Stronger Communities Draft ‘Theory of Change’ for the evaluation Steve Connelly University of Sheffield 1 st February 2018

Introduction This slide pack sets out a draft ‘theory of change’ for the CMF-funded projects within RBSC. It is intended to • be the basis for a structured approach to the evaluation of the programme • prompt discussion about what can and should be evaluated, and how • lead to better understanding of how the projects might work, where there might be synergies and overlaps and potential conflicts, and identify previously unpredicted outcomes (good and bad) • identify areas of lack of clarity for further discussion and development

Structure • key points and outstanding questions and issues • logic models (LM), theories of change (To. C), and the context/mechanism/outcome approach • BSC’s 9 types of mechanism, and relationship to projects and outcomes • an example of a To. C for one mechanism • the full To. C • an example of individual project logic models

key points • it’s very complex – even ‘simplified’ down to 9 linear ways of having effects (‘mechanisms’) • most mechanisms have multiple outcomes, so we have an ‘attribution problem’ – which projects caused what? • the project deliverables are focused very much on activities, less on the processes by which they will have effects (capacity change (8), behaviour change (10), direct benefits (7)) • of the 16 deliverables related to wider benefits, only 5 are attributed to projects • the (understandable) project focus on ‘doing stuff’ with communities leaves out tackling contextual factors in both communities and state/3 rd sector bodies: how do we ensure support for enduring change both ‘culturally’ and in material terms?

outstanding questions/issues • how can the complexity of the project be presented in ways which enable informed partner input? • we need more measures of if/how activities are having effects, and how these lead to benefits • what (if any) evidence do we currently have to support the causal links and assumptions? • what differences might there be between groups in terms of how relevant/accurate the assumptions are? and so of how effective the intervention is? • what risks are associated with the assumptions? what measures can be taken to mitigate them? • how do we examine each activity for its sustainability?

Logic models logic models set out the logic of an intervention: how it is supposed to have effects Activities/ outputs Capacity change Behaviour change Direct benefits Wider benefits Adapted from Mayne, J (2015) ‘Useful Theory of Change Models’, Canadian Journal of Program Evaluation, 30 (2) 119 -142

Logic models + assumptions become theories of change • a full ‘theory of change’ (To. C) includes the ways in which the logic model stages are assumed to connect. • these assumptions also point to conditions which have to be present if the LM is to be made real (i. e. if the intervention is to work) and suggest alternative outcomes Activities/ outputs Assumptions about the basic purpose and function of the activity Capacity change Assumptions about reach, how capacity is changed, and conditions required Behaviour change Assumptions about how capacities lead to behaviour change, and conditions required Direct benefits Assumptions about how behaviour change benefits the individual Wider benefits Assumptions about how direct benefits lead to wider social benefits

mechanism + context → outcome • evaluation theorists also use a language of ‘mechanism’ for the process which leads from activity to benefits, and the idea that mechanisms act in contexts to lead to outcomes: the same mechanism in a different context might lead to different outcomes • here ‘mechanism’ means the first three stages of the LM. . . and ‘outcomes’ are the benefits Activities/ outputs Capacity change Behaviour change Direct benefits Wider benefits

9 types of mechanism identifiable across the projects Working with key individuals Providing advice Training/education Getting individuals together across communities Environmental projects Financial support for community groups Targeted state support to families and young people Increasing enforcement activity Working with governance organisations (state and VCF)

9 types of mechanism, 8 partners, 24 projects RMBC: ACE RMBC: CYPS RMBC: RE Clifton Learning Partnership REMA RUCST Premier Learning Kimberworth Park Community P’ship Working with key individuals Providing advice Training/education Getting individuals together across communities Environmental projects Financial support for community groups Targeted state support to families and young people Increasing enforcement activity Working with governance organisations (state and VCF) 2 3 4 1 1 1 4 5 6 8 9 3 1 1 2 3 4 5 1 1 1 2 2 4 5 6 8 1 2 3 3 1 2 9 0 5 6 7 1 1 0 1 1

mechanisms in detail (1) Activities/ outputs Capacity change Behaviour change Street champions Increased engagement skills, confidence, knowledge to - provide help to others - engage with the state/VCF - mediate disputes Engagement with others in the community Reporting problems to relevant governance organisations Drawing up a ‘Residents’ Charter’ Advice services Increased knowledge of rights, services available More, faster, earlier, appropriate accessing of services Training/educ ation ESOL, and other skills/knowledge taught along with ESOL Better knowledge of English Better understanding of rights/responsibilities Other knowledge/skills (with vocational etc. relevance) More engagement with service providers More exercising of democratic rights More socially responsible action within communities Move into employment Getting individuals together across communities ESOL, Citizenship Project, BSC Forum, festivals/events, sport, Love is Louder events Better understanding, trust, respect for different Others Less antagonistic behaviours Enforcement of norms of tolerance within communities (e. g. stopping/reporting hate crime) Self-organised inter-community collaborative working Better school attendance Working with key individuals Providing advice Community navigators Good neighbour project

mechanisms in detail (2) Activities/ outputs Capacity change Environmental projects CLP environmental projects Provided with skills and opportunity to deliver projects More respect amongst participants and others for the immediate environment Financial support for community groups Giving small grants to community groups/organisations Providing support to groups to bid miscellaneous Targeted state support to families and young people Outreach/street working with young people Targeted bespoke family support YP more confident and selfaware as citizens Better (state) understanding of YPs’ issues Reduced ASB (including drugs) Collective activities Participation in youth forum Increase in education participation rates Reduction in family abuse Increasing enforcement activity Reducing CTax fraud Enforcing housing standards Enforcing environmental regulations - fly tipping Understanding of risks and penalties for noncompliance/lawbreaking Staff and IT (surveillance, reporting) increased Cessation of illegal activity Compliance with rules BSC Forum? Better understanding of communities Trust, understanding, skills to work in partnership Better knowledge of possible funding/resource streams More appropriate service provision More effective partnership working: reduced duplication, signposting between organisations, new joint actions More external resources successfully bid for Working with governance organisations Behaviour change Direct improvements: litter cleared, allotments, planting Less littering in future More food growing Better school attendance miscellaneous

mechanisms and outcomes Mechanisms Working with key individuals Providing advice Direct benefits Quicker settling in More effective resolution of service problems Training/education More harmonious community life (less involvement in racial abuse etc. , disputes resolved) Getting individuals together across communities More harmonious personal lives (within families, at school, less friction with authorities) Environmental projects Financial support for community groups Targeted state support to families and young people Increasing enforcement activity Working with governance organisations (state and VCF) Satisfaction/wellbeing from social engagement/participation More confident and able to exercise rights/responsibilities (Better) employed/higher income Living in better physical environment (greenery, litter, fly tipping…) More secure tenancy in better housing

direct and wider benefits Direct benefits Quicker settling in More effective resolution of service problems More harmonious community life (less involvement in racial abuse etc. , disputes resolved) More harmonious personal lives (within families, at school, less friction with authorities) Satisfaction/wellbeing from social engagement/participation More confident and able to exercise rights/responsibilities (Better) employed/higher income Living in better physical environment (greenery, litter, fly tipping…) More secure tenancy in better housing Wider benefits More appropriate service provision More appropriate calls on services More participation in ‘governance’ broadly Positive changes in state (etc. )/resident behaviour Less hate crime and other ASB More meaningful interactions between different groups Better landlord/tenants relationships Positive changes in resident/resident behaviour and relationships Better neighbourhood environment Better housing (physical, security) Wealthier community Higher (mental/physical/ emotional) well-being Improved conditions for (deprived) communities

theory of change for one mechanism • Getting individuals together across communities Behaviour changes: Less antagonistic behaviours Enforcement of norms of tolerance within communities (e. g. stopping/reporting hate crime) Self-organised intercommunity collaborative working Better school attendance Activities: ESOL, Citizenship Project, BSC Forum, festivals/events, sport, Love is Louder events Capacity changes: Better understanding, trust, respect for different Others Antagonism between communities, and towards the state for perceived biases/failings, comes from a lack of understanding of, and empathy with, different Others, and direct contact will address this (rather than worsen it) People who engage are those whose positions need changing Contact provides opportunities for effective, meaningful dialogue, and learning about the Other – and not for increased antagonism/reinforcing of prejudices. Sufficient, competent staff are involved, who can (at minimum) effectively deal with tension/friction, and are able to facilitate effective (potentially very difficult) interactions Direct benefits Individuals are sufficiently confident to act on new understandings within their cultural context, and robust enough for this to be sustained Opportunities and support are available for further engagement Local and national policy, the media and wider societal discourses, do not provide overriding counter-messages Wider benefits Behaviour changes are enduring Skills are successfully deployed (personal and mediation etc. ) Communities respond positively to changed behaviours

the full To. C: mechanisms and assumptions Mechanisms Activities Working with key individuals Street champions Community navigators Good neighbour project Capacity change Increased engagement skills, confidence, knowledge to - provide help to others - engage with the state/VCF - mediate disputes Behaviour change Engagement with others in the community Reporting problems to relevant governance organisations Drawing up a ‘Residents’ Charter’ Assumptions Tacking problems requires communication (at minimum) between state and residents, and key individuals can provide routes into the community which the state (and VCF? ) cannot achieve themselves, due to problems of trust, legitimacy, language, resource capacity etc. Advice services (REMA and CLP) ignorance is a problem - people would value services/rights but fail to take them up or act on their rights because they don’ know what is available, how to access them etc. or what their rights are. ESOL, and other skills/knowledge taught along with ESOL Suitable individuals can be identified, and will keep with the programme Provider organisations are trusted enough for relevant people to engage with them Programmes will deliver the required skills and knowledge Participants will be willing and able to act Participants will share the programme goals and act appropriately Participants will have the networks and legitimacy to act Cultural norms support increasing engagement with the state Service providers will respond and reinforce behaviour Increased knowledge of rights, services available Those in need will know about the services - and have the confidence and ability (mobility, language) to access the service Appropriate and accurate advice will be available, from people with the skills/attitudes/knowledge to engage with the target population Better knowledge of English Better understanding of rights/responsibilities Other knowledge/skills (with vocational etc. Relevance) More, faster, earlier, appropriate accessing of services People will have the confidence and resources to act on the knowledge Cultural norms support increasing engagement with the state Appropriate services will in fact be available and accessible and able to meet any increased demand People lack knowledge and skills needed to play a full part in society and they see education/training as a way of addressing this Those in need will know about what is available - and have the confidence and ability (mobility, language) to access the service People will be able to attend regularly enough to learn Sufficient high quality training, appropriate to users’ needs, is available Providing advice Assumptions Training/education Assumptions More engagement with service providers More exercising of democratic rights More socially responsible action within communities Move into employment Education/knowledge will empower people Cultural norms support increasing engagement with the state Service providers are willing/able to engage and meet increased demand Governance organisations are willing/able to engage with democratic expression Getting individuals together ESOL, Citizenship Project, BSC Forum, Better understanding, trust, respect for different Others Less antagonistic behaviours across communities festivals/events, sport, Love is Louder Enforcement of norms of tolerance within communities (e. g. events stopping/reporting hate crime) Self-organised inter-community collaborative working Better school attendance Assumptions Antagonism between communities, People who engage are those whose positions need Individuals are sufficiently confident to act on new understandings and towards the state for perceived changing within their cultural context, and robust enough for this to be biases/failings, comes from a lack of Contact provides opportunities for effective, meaningful sustained understanding of, and empathy with, dialogue, and learning about the Other – and not for Opportunities and support are available for further engagement different Others, and direct contact increased antagonism/reinforcing of prejudices Local and national policy, the media and wider societal discourses, will address this (rather than worsen Sufficient, competent staff are involved, who can (at do not provide overriding counter-messages it) minimum) effectively deal with tension/friction, and are able to facilitate effective (potentially very difficult) interactions

Mechanisms Environmental projects Activities CLP environmental projects Assumptions Antagonism and dissatisfaction with neighbourhoods related to poor environmental condition, and social and physical improvements can be achieved if people improve the environment themselves Capacity change Provided with skills and opportunity to deliver projects More respect amongst participants and others for the immediate environment People with enthusiasm can be recruited and retained Behaviour change Direct improvements: litter cleared, allotments, planting Less littering in future More food growing Better school attendance Project outputs can be maintained (and not vandalised) – either by the state or by the community Improved environment encourages more environment-friendly behaviours Working with governance BSC Forum? organisations (state and VCF) More appropriate service provisions More effective partnership working: reduced duplication, signposting between organisations, new joint actions More external resources successfully bid for Assumptions Better understanding of communities Trust, understanding, skills to work in partnership Better knowledge of possible funding/resource streams Service provision is currently weakened by governance Service providers – as organisations and issues, in particular around integrated/partnership individuals – are able and willing to learn working, and (lack of ) understanding of relevant Governance organisations are willing to communities (in particularly Roma) work together Service providers are adequately resourced to act on new understanding Governance organisations are able to work together (adequate resources/support available) Increasing enforcement activity Reducing CTax fraud Enforcing housing standards Enforcing environmental regulations - fly tipping Understanding of risks and penalties for non-compliance/lawbreaking Staff and IT (surveillance, reporting) increased Cessation of illegal activity Compliance with rules Assumptions Violations of environment/housing regulations play a significant role in creating antagonism between communities, and stricter enforcement can be done effectively without antagonising communities/undermining trust in the state Calculations of costs/benefits of illegal/no-compliant activity favour cessation and compliance (may be affected by perceptions of risks Provision of services to enable compliance (e. g. of waste disposal, support for housing repairs) Targeted state support to families and young people Outreach/street working with young people Targeted bespoke family support People beyond those directly affected know about enforcement action and the risk of it People ‘read’ enforcement in terms of upholding law, not as oppression Intelligence – from IT and staff/community reporting – can be effectively and swiftly acted upon Better (state) understanding of YPs’ issues Assumptions Financial support for community organisations/groups Assumptions Giving small grants to community groups/organisations Providing support to groups to bid Appropriate groups can be reached Small financial barriers are important in preventing groups achieving aims Lack of capacity/knowledge/confidence in bidding is a significant barrier Reduced ASB Collective activities Participation in youth forum Increase in education participation rates Reduction in family abuse (Reduction in drug use? )

the full To. C: outcomes and assumptions Direct benefits (Better) employed/higher income Assumptions: Language skills and confidence are barriers to employment Relevant and accessible jobs exist Better physical environment (greenery, litter, fly tipping…) Assumptions: Sufficient numbers of people participate, and continue to participate/act Other members of community and service providers support actions More effective resolution of service problems Assumptions: Service providers are willing/able to respond to demand More harmonious community life (less involvement in racial abuse etc. , disputes resolved) Assumptions: Behaviour changes are enduring Skills are successfully deployed (personal and mediation etc. ) Communities respond positively to changed behaviours More secure tenancy in better housing Assumptions: Landlords are able/willing to make investments Tenants are able/willing to be ‘good tenants’ – longer lets, less damage etc. Quicker settling in Assumptions: New arrivals are reached, and are willing/able to respond Service providers are willing/able to respond to demand Satisfaction/wellbeing from social engagement/participation Assumptions: Engagement is in fact positive (and not stressful etc. ) Benefits are significant in relation to other causes of ill-being More confident and able to exercise rights/responsibilities Assumptions: Participation and learning are sufficient to enable active engagement Service providers, governance organisations, other members of community are receptive Opportunities for engagement exist and are accessible More harmonious personal lives (within families, at school, less friction with authorities) Assumptions: Service providers, governance organisations, other members of community are receptive and supportive.