Underrepresented Communities in Higher Education A Realist Evaluation

Underrepresented Communities in Higher Education: A Realist Evaluation Dr Adam Formby (a. formby@yorksj. ac. uk) York St John University Twitter add: @ad_formby

Introductions Dr Adam Formby Research interests: sociology and social policy of Youth, access-to-HE, inequalities, stigma and identities, graduate transitions, unemployment and precarity, work and social policy analysis Current Projects: - Elimination of jobs in UK Youth labour markets (Ralston and Formby, 2020). People, Place and Policy Special Issue on Youth and Precarity (Devany et al, 2020). - Realist Evaluation of ‘Stepping up’ Youth Work Initiative with North Yorkshire Police in York (as well as supporting general evidence-based approaches) - Supporting underrepresented groups into HE across West and North Yorkshire through Widening Participation in the Community – a Realist Evaluation (Formby et al, 2020 a; Formby et al. 2020 b)

Outline 1. Evaluating access-to-Higher Education initiatives in the UK (what we term ‘widening participation’) 2. Go Higher West Yorkshire Uni Connect 3. Realist Evaluation of Community Initiatives a. Formby, A. Woodhouse. A and Basham, J. (2020) Reframing widening participation towards the community: a realist evaluation. Journal of Widening Participation and Lifelong Learning, 22 (2). pp. 184 -201. b. Formby, A. Woodhouse, A. Basham, J. and Roe, F. (2020) ‘A Presence in the Community’: Developing Innovative Practice through Realist Evaluation of Widening Participation in West Yorkshire, Journal of Widening Participation and Lifelong Learning, 22 (2). pp. 184 -201

1. Evaluating Access-to-Higher Education initiatives in the UK

The Access-to-HE UK Landscape • Since 2010, all universities in the UK have been required to implement widening participation (WP) strategies • Ensuring students from underrepresented communities can access higher education (HE) (Harrison and Waller, 2017). • £ 832 million spent annually by HE sector on such initiatives • UK HE: deeply socially stratified (Hayton and Bengry. Howell, 2016) in HE participation rates regarding socially underrepresented young people (Tazzyman et al. , 2018).

Three Components of WP in UK 1. Access-to-HE (targeted measures at secondary school, college and the wider community) 2. Inequalities within HE (e. g. lived experiences of student at University) 3. Entering graduate labour market e. g. supporting underrepresented graduates in transition to work (see Formby, 2017)

So what does access-to-HE WP look like? • Modern WP looks to create different types of activities that support learners throughout their educational trajectories: • • Careers focused events, Informational events about HE (finance events) One-to-one mentoring Personal statement workshops, mentoring, student finance Talks Parental engagement and residentials Campus visits

Establishment of Uni Connect (formerly NCOP) • Significant demand by UK government to improve methods of evaluation of WP activity at local and national levels (Department for Education, 2019: 77) • Establishment of Uni Connect in 2017 • Partnerships that aimed to Partnerships to ‘deliver outreach programmes to young people in years 9 to 13 (aged between 13 and 18 in schools and colleges)’, • Analysis identified a gap between average GCSE results and expected progression to HE (Crawford et al. , 2016) • ‘Gaps’ were observable at the ‘ward’ level (HEFCE, 2016), emphasising the extent access-to-HE is mediated through forms of social and cultural inequality (Crawford et al. , 2016: 570).

‘Evaluating’ Widening Participation "We want you to build in evaluation of your access measures right from the start so you can maximise the effectiveness of your efforts. We appreciate that evaluating the outcomes of access activities is not always easy but it is vital if we are to improve understanding of what works best, share best practice across the sector and demonstrate to Government the value of investment in this area” (LES EBDON, FOREWORD TO HOW TO PRODUCE AN ACCESS AGREEMENT FOR 2014/15, OFFA)

‘Capturing’ and ‘Demonstrating’ Impact • Current widening participation strategies aim to be ‘smarter’ (Gov. uk, 2018) • Support activities which have ‘demonstrable impact’ in improving access and participation, progress in higher education and employment outcomes (Gov. uk, 2018) • Challenges • Different cultures and practices around WP • What works in different ways for in different institutions/communities – and how does it do so? • Implementing new initiatives into schools and FE colleges

2. Go Higher West Yorkshire Uni Connect

Go Higher West Yorkshire Uni Connect • Engaged in a Realist approach re: evaluating NCOP across Leeds, Bradford and Wakefield set up by GHWY, Dr Jemma Basham and Prof. Ray Pawson • Capturing ‘Complexity’ of WP practice and policy in differentiated underrepresented communities across the region • Recent years have seen a growth in ‘realist’ methods in WP research (Thomas, 2017) “Takes account of complexity, and is concerned with making sense of context, mechanisms and outcomes[…]thus more appropriate for this issue where the topic (WP and fair access) and the context (higher education institutions) are both complex issues with multiple understandings, levels and functions” (Thomas, 2017: 4)

Why Realist Evaluation? • Recent years have seen a growth in realist research methods in WP research (Thomas, 2017) “Takes account of complexity, and is concerned with making sense of context, mechanisms and outcomes” (Thomas, 2017: 4). • Committed to arguments that programs engage with real problems • Method neutral • Attempts to inform realistic development in policy making for those who run programs and for participants who those programmes target • An effective way of explaining existent ‘gaps’ at ward level (Pawson and Tilley, 1997)

A Foundational Scientific Realist Philosophy A scientific model of explanation, which avoids the traditional epistemological poles of positivism (deductive – hypothesis building and testing) and relativism (inductive – ‘how people frame the world). Pawson and Tilley, 1997

Mechanics of Explanation • We go beyond ‘what works? ’ to thinking about ‘what works for whom? And in what circumstances and what in respect – and how’ • Mechanisms = Refers to the underlying social or psychological drivers that cause the reasoning of actors (reasoning(s) and resources) • Contexts = institutional. Key as context(s) influences ‘reasoning’ and, secondly, generative mechanisms can only work if circumstances are right • Outcomes = Patterns of outcomes (intended or unintended) Marchal et al (2017)

Outcome = Mechanism + Context (C) (X) intervention Mechanism (M) Patterns of Outcomes (O)

Why it is so effective for WP evaluation? Contextual Explanation • Focus on context(s) that allows RE to be extensively effective in analysing a wide range of WP programmes – especially in the community space • Pawson (2018: 212) notes how ‘contexts’ in RE include: Locations (spatial and geographical characteristics), Individuals (e. g. training) Relationships Institutional arrangements (e. g. organisational arrangements) • The wider infrastructure (e. g. wider social, economic and cultural contexts) • • • Making RE an effective way of explaining existent ‘gaps’ at ward level

3. Realist Evaluation of Community Initiatives

Community interventions • Two parts 1. WP in the Community 1. 2. 3. WP has been moved to the community space through the introduction of HEPs (Higher Education Providers) who facilitate WP community outreach. Middle-range theory: sense of place Methods: realist interviews with outreach staff 2. WP in schools and colleges 1. 2. 3. Outreach staff in schools and colleges to open up ‘spaces’ for WP and shift culture Middle-range theory: sense of place Methods: realist focus groups with HEPO staff and realist interviews with management

Interventions in the Community • Such approaches consider different contexts – help elucidate how WP is working in distinct community spaces (Lumb and Roberts, 2017: 22). • Furthermore, by positioning WP in the community, there is increased scope for embedded ‘sense of place’ (Cresswell, 2009). • Pretty et al. (2003: 274) argue that ‘sense of place’ “emerges from involvement between people, and between people and place’’. • By tapping into the social relations that learners inhabit daily, it has been found that WP has the potential to be transformative in the support it offers learners to access HE (Scull and Cuthill, 2010).

Interventions in the Community Examples

Interventions in the Community Examples • https: //www. gohigherwestyorks. ac. uk/impa ct/ • bespoke and community focused activities tailored to the needs of the young people and communities • We run and set up and running informal activities explicitly positioned separate to formal school or college settings, such as: • Creative/art focused projects; • Individual interventions (e. g. one-to-one mentoring); liaison with local community groups (to engage with parents and carers) • Setting up information stalls in local community spaces and working with local youth groups.

Community WP • Establishment of ‘sense of place’ (Cresswell, 2009) ensured an effective platform for Officers to innovate WP in different ways. ‘Paved the way’ for more serious conversations (built trust) about HE choices and future trajectory - continuously reinforced as a mechanism to build relationships: Sometimes you can have an outrageous activity…however that can start initiating conversations you need to have and sometimes in the community if you go in with this full-on HE session about student finance, people are not going to turn up (Outreach Officer based in Leeds). • Importance of ‘informal relationships’ also stressed for individualised interventions (e. g. one-to-one mentoring), especially in terms of building trust with learners: Its quality…not just quantity […] just by having these smaller groups or just those one-to-one relationships you can really build up a rapport (Outreach Officer based in Wakefield). • Overall, RE emphasised how important it is to avoid generalised one-size-fits-all approaches that do not account for different contexts (Lumb and Roberts, 2017; Pawson and Tilley, 1997) • …. But also how challenging community WP is to do outside of ‘formal’ institutions (e. g. schools and colleges)

2. WP in schools and colleges: A Realist Evaluation • Exploring how Higher Education Progression Officers (HEPOs) (dedicated staff who support GHWY’s WP activity within NCOP target schools and colleges) normalise WP, in reflection of their community context and space • Realist Focus groups and interviews with HEPOs and Management across schools and colleges in West Yorkshire

GHWY HEPO staff : A Community driven Approach • HEPO staff: “boundary spanners” (Sandholtz and Finan, 1998) that facilitated broad-ranging partnerships with GHWY consortia • Similar to ‘transition learning mentors’ - directing support for young people that fit widening participation criteria (Woods et al, 2010) • Community focused approach embedded in GHWY provision, • Helps identify different aspects of need in schools and college settings across West Yorkshire. • Supposed to give the wider community a sense of ownership in that they are contributing to ‘progression’ based initiatives (Scull and Cuthill, 2010). • Relationships established through community focused WP – and built over the long-term are effective at establishing trusting relationships with students (Into. University, 2017: 27)

A Conceptual Framework for Realist Evaluation of HEPO Role

Middle-Range Theory: ‘Sense of Place’ (Cresswell, 2009) • Sheds light on how the role of HEPOs was established so thoroughly (Cresswell, 2009: 2), • Provides some causal explanation regarding conditions where the HEPO role worked and where it could be expanded in future. • Schools and colleges are ‘activity spaces’ – they comprise spatial networks, connections and locations “within which a particular agent operates” (Massey, 1994: 55). • For HEPOs, familiarity with institutional sites is necessary to organise WP activity and embed the notion of ‘progression’ in the social relations of that institutional space. • Important distinction is made regarding • The introduction of physical resources and WP activity • And the ‘sense of place’ associated with community and social norms that enabled HEPO staff to enhance WP provision (Cresswell, 2009).

Sense of Place • HEPOs familiarity with ‘sense of place’ triggered four specific ‘programme mechanisms’ 1. ‘Sense of place’ increased volume of WP provision 2. ‘Sense of place’ resulted in more targeted WP 3. ‘Sense of place’ increased organisational familiarity 4. ‘Sense of place’ normalises the HEPO role in institution (shifted cultures)

Outcome 1 – Context(s) where HEPO Strategy Worked • Increased volume and bespoke delivery: Out of the three schools I have worked in, two have had NCOP staff and there has been a significant difference in outcomes. There has been better partnership working, better results in terms of participation and more bespoke delivery (Social Inclusion Manager at College). • Opening spaces for WP provision (such as knowledge gain; increased contact time and visibility of WP more broadly): Right at the start when I asked my student to fill in their survey, they said that they constantly get information and this is shoved down their throats - and [for us] ‘we feel like we’re coming in from a new perspective - not just another person there to shove stuff down their throat’ (Focus Group - HEPO).

Outcome 2 – Context(s) where HEPO strategy worked less • Less ‘sense of place’, less HEPO engagement (also at organisational and institutional levels) Was only two months ago that my head teacher finally realise that I don’t have to only work with sixth form, he just assumed that I only work with sixth form…[but] now that I’ve got that link with that senior management. . . we have this kind of regular meeting every three weeks, he’s really excited about it (Focus Group - HEPO). • Societal/community contexts outside of school or college (where ‘sense of place’ programme mechanisms do not ‘fire’) It’s a barrier with a lot of parent sometimes, if they come from like…a real working class background and then you’re sort of like putting in the idea that the head of like you know, the kid can go into university, it’s as though you they can sort of see that as an attack on them, it’s being like, “Oh, they have not done so, therefore they are not good enough” (Focus Group - HEPO).

HEPOs – Need to establish ‘Sense of Place’ in the Community Setting? • Avoiding ‘deficit model’ approaches (Holdsworth, 2009) through opening WP provision to the community setting: So like stuff to come to the school, it’s something that perhaps we [could] focus on – like within the colleges, going to community centres and all that. That sort of angle, of getting embedded in the community, so it’s not the bulk of our job, but it is something I think most of us think would find really useful to do (Focus Group - HEPO)

Conclusion – RE: ‘What happens on a sunny day in Whitby is not the same on a rainy day in York’ • HEPOs – even when ideal institutional causal conditions exist to ensure that HEPOs sufficiently embed ‘WP’ practice • limitations relate to wider contextual forces • Outreach – effective when community (and families) can ‘buy-in’ in relation to ‘progression’ (Scull and Cuthill, 2010). • HEPOs involvement with perceptions and attitudes on HE in the community space (and institutional context) is an additional avenue to explore in future

Outcome: Helping underrepresented communities to 1 progress to higher education is Resources (GHWY) most effective through engaging 2 3 Negative Positive Time taken to set up multiple interventions Staff Disruption (usually at the beginning of initiatives) Expert Resources Extra WP Interventions Tailored focus on institutional space and community multiple interventions in varying relevant spaces at different points in OUTCOMES Tailored support for young people and parents in the institutions and the wider community Negative Positive Distrust of HE Challenges in engaging communities in the wider community Institutional impact Emergent Cultures of WP Normalisation of WP Further external collaboration Reasoning the student life-cycle. Interventions need to take a holistic, respectful and community-based approach. Further, by tailoring support in relation to the space that young people inhabit allows for a stronger culture of WP to take place in underrepresented communities. Context 1. The school or college (sense of place) 2. The wider community (outside of the school or college) 3. The ward/locality? ?

Conclusion – RE: ‘What happens on a sunny day in Whitby is not the same on a rainy day in York’ • Mutually reinforcing approaches: To achieve positive outcomes – need both institutional and communitybased WP – see logic model • Sense of Place (Middle-range theory) • Realist Evaluation • Focus on contextual explanation • An extremely useful approach to evaluate WP • Always comes back to: ‘What works for whom? And in what circumstances and what in respect – and how’ “Importance of engaging the community space and the situated circumstances where WP works. This allows for in -depth and nuanced explanations about the efficacy of WP” (Formby et al, 2020: 11)

References Realist evaluation and WP • Formby, A. Woodhouse. A and Basham, J. (2020) Reframing widening participation towards the community: a realist evaluation. Journal of Widening Participation and Lifelong Learning, 22 (2). pp. 184 -201. • Formby, A. Woodhouse, A. Basham, J. and Roe, F. (2020) ‘A Presence in the Community’: Developing Innovative Practice through Realist Evaluation of Widening Participation in West Yorkshire, Journal of Widening Participation and Lifelong Learning, 22 (2). pp. 184 -201 • Thomas, L. (2017) Understanding a whole institution approach to widening participation: Final report, Office for Fair Access • Harrison, N. (2018) Why Evaluate? https: //www. sheffield. ac. uk/apse/wp/whyevaluate/harrison General Resources • Marchal, B. van Belle, S. Westhorp, G. Peersman, G. (2017) Realist Evaluation (http: //www. betterevaluation. org/en/approach/realist_evaluation) • Pawson, R. & Tilley, N. (1997). Realistic Evaluation. London: Sage. • Pawson, R. & Tilley, N. (2001). Realistic Evaluation Bloodlines. American Journal of Evaluation 22: 317324. http: //aje. sagepub. com/content/22/3/317. full. pdf+html • Pawson, R. (2013). The science of evaluation: a realist manifesto, London, SAGE Publications. Other recommended reading • Forster, N. Dalkin, S. Lhussier, M. Hodgson, P. Carr, S. (2016) Exposing the impact of Citizens Advice Bureau services on health: a realist evaluation protocol, BMJ: 6. • An introduction to realist evaluation including a downloadable chapter from Pawson and Tilley (1997) (reproduced with permission from the authors) www. communitymatters. com. au/gpage 1. html • Pawson, R. & Manzano-Santaella, A. (2012). A realist diagnostic workshop. Evaluation 18: 176191. http: //dx. doi. org/10. 1177/1356389012440912 • Marchal, B. , Van Belle, S. , Van Olmen, J. , Hoerée, T. & Kegels, G. 2012. Is realist evaluation keeping its promise? A literature review of methodological practice in health systems research. Evaluation, 18, 192 -212. • Astbury, B. & Leeuw, F. (2010). Unpacking Black Boxes: Mechanisms and Theory Building in Evaluation. American Journal of Evaluation 31(3): 363 -381. http: //www. itg. be/internet/ds/tde/doc/Astbury%20%26%20 Leeuw%20. pdf

Mechanics of Explanation - CMOCs • Programme Theories - “ascertain the rationale regarding the assumptions of stakeholders around how a programme works” (GHWY, 2018: 6) at onset of the initiative. • As such, the realist evaluation looks to assess how and why these programme theories achieve their respective aims (Marchal et al, 2018: 83). • Aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context • Mechanisms = Refers to the underlying social or psychological drivers that cause the reasoning of actors • Contexts = institutional. Key as context(s) influences ‘reasoning’ and, secondly, generative mechanisms can only work if circumstances are right • Outcomes = Patterns of outcomes (intended or unintended) Marchal et al (2017)

Bringing Together To. C and RE?

Appendices: RE Example – Does ‘CCTV’ Reduce Crime? A local car park has recently had vehicles regularly broken into and several personal items stolen. Police respond through implementing CCTV around the car park leading to much reduced crime in the area Why do you think CCTV has reduced crime in this instance? (Adapted from Pawson and Tilley, 1997)

Does ‘CCTV’ Reduce Crime? • No! On it’s own – CCTV does not do anything • But it may lead to a chain of ‘reasoning and reaction(s)’ that explain an outcome such as: • A ‘You’ve been framed’ Mechanism: increased awareness of ‘risk’ puts off future criminals? • A ‘Memory Jogging’ Mechanism: Car owners, on seeing the CCTV, remember to lock doors and secure vehicles • A ‘Nosey Parker’ Mechanism: on seeing CCTV, more people use the car park raising attendance which enhances natural surveillance (deterring ‘would be’ offenders) • And so on… (Adapted from Pawson and Tilley, 1997)

…. Theory of Change is essentially a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context. Image credit: Sidney Harris Theory of Change – A Definition …Focused in particular on mapping out or “filling in” what has been described as the “missing middle” between what a program or change initiative does (its activities or interventions) and how these lead to desired goals being achieved Theory of Change (2019)