Evaluating and assessing the impact of working with

Evaluating and assessing the impact of working with young people in regional and minority language revitalisation Rhys Jones, Elin Jones (Prifysgol Aberystwyth) and Fiona O’Hanlon (Edinburgh University)

Strwythur/Structure • Cyd-destun allanol: gwaith • ieuenctid mewn iaith leiafrifol • Cyd-destun mewnol: heriau • a chyfleon • Theori newid • Cymorth a chamau nesaf • • External context: youth work in a minority language Internal context: challenges and opportunities Theory of change Support and next steps

Cyd-destun allanol/External context • Cyd-destun allanol 1 • External context 1 • Polisi ar sail tystiolaeth • (Test, Learn, Adapt) • Mae ymyrraethau’n methu • yn aml • • Mae angen cyfiawnhau gwariant cyhoeddus Evidence-based policy (Test, Learn, Adapt) Interventions often fail The need to justify public spending

Cyd-destun allanol/External context • Cyd-destun allanol 2/External context 2 • “The strategy recognises. . . the need to strengthen the evidence base on the impact of youth work across Wales. ” • “much more needs to be done to systematically and robustly evidence the contribution of youth work” (both from the National Youth Work Strategy for Wales 201418). • Youth Work Reference Group (2015): “to develop and implement a National Outcomes Framework”.

Cyd-destun allanol/External context • Cyd-destun allanol 2/External context 2 • “Self-evaluation processes use the triangulation of evidence-based information and data, people’s views and direct observation of practice to inform outcomes and measure impact on young people. In order to plan improvement effectively we need to know how well we are doing” (Scotland’s National Youth Work Strategy 2014 -19), p. 17). • Measuring impact as one of 5 principles of youth work

Cyd-destun allanol/External context • Cyd-destun allanol 2/External context 2 • Develop efficient and user-friendly ways of collecting and analysing quantitative and qualitative data at local, regional and national level to inform service planning, delivery, assessment and continuous improvement (Ireland’s National Youth Strategy 2015 -20, p. 35) • “Undertake and publish research that demonstrates the positive role of youth work in achieving national outcomes and local priorities” (National Youth Council of Ireland’s Strategic Plan, p. 23).

Cyd-destun allanol/External context • Cyd-destun allanol 3/External context 3 • “Continue to build on our evidence about the Welsh language and its speakers, as a basis for assessing the effectiveness of our interventions and developing Welsh language policy. • Considering, as programmes are being planned, the most appropriate methods of evaluating their effect. • By combining qualitative and quantitative data we aim to enhance our understanding of speakers’ linguistic experiences and practices as they move through different periods of their lives” (Cymraeg 2050, p. 77).

Cyd-destun allanol/External context • Cyd-destun allanol 3/External context 3 • “In order to track progress on the Plan, Bo rd na Ga idhlig will collate and publish information from a range of sources, including public authorities, Gaelic organisations and community groups in an annual National Gaelic Language Plan progress report. • Increasing the use of Gaelic is at the core of the National Gaelic Language Plan and ways of measuring this will be developed to give a clear picture as to how the strategies for increasing use are working” (National Gaelic Language Plan 2018 -23, p. 55).

Cyd-destun allanol/External context • Cyd-destun allanol 3/External context 3 • Local Action Plans “supplemented by a language audit process which will identify ongoing strengths and weaknesses and provide time-series evidence of the impact which language-related policies and reforms would have on actual language use” (20 year Strategy for the Irish Language, p. 24). • Overall… – – An emphasis on the need to demonstrate outcomes and impact Coming from funders and practitioners Couched in terms of effectiveness rather than value for money New methods, synthesis, a focus on time

Cyd-destun allanol/External context • • Beth yw’r cyd-destun allanol ar gyfer mesur canlyniadau a thraweffaith eich gwaith? • Beth yw’r cyd-destun • pwysicaf (gwaith ieuenctid neu waith iaith)? • Sut mae’r angen i • arddangos canlyniadau yn dylanwadu ar y berthynas gyda noddwyr? What is the external context for evaluating the outcomes and impact of your work? What is the most important context (youth work or language work)? How does the need to show outcomes influence relationships with sponsors?

Cyd-destun mewnol/Internal context • “Urdd Gobaith Cymru is a National Voluntary Youth Organisation with over 53, 000 members between the ages of 8 – 25 yrs old. It provides Sport, Cultural, Community and Residential opportunities through the medium of Welsh for children and young people in Wales that will enable them to make a positive contribution to the community. The Urdd has nurtured generations of young men and women to be proud of their country, open to the world and living embodiments of our language and culture, along with the universal values which we cherish in Wales. Its contribution to…the lives and confidence, and mental health of our young people over the years has been immense” (Urdd Gobaith Cymru Annual Report 2017 -18)

Cyd-destun mewnol/Internal context • “Gael Linn is a national organisation whose principal aim is to foster and promote the Irish language and its heritage as a living language and as an expression of identity at policy and at community level. ” • Feachtas is a young Irish-language organisation that was established in 1980. Objectives of the organisations: – a) To promote the Irish language among young people – b) To help the social and personal development of a young person – c) Help to build an Irish-language community • Cumann na bh. Fiann provides opportunities for young people to meet through the medium of Irish and create networks of Irish speaking friends throughout the country

Cyd-destun mewnol/Internal context • Comann na Gàidhlig organises a range of sporting and cultural activities for Gaelic-speaking young people. It aims to: – create opportunities for them to use Gaelic out with the classroom so that young people can use Gaelic in informal settings along with their peers. – increase the young people’s confidence in using Gaelic and their fluency. – Research shows young people, parents and teachers feel local community initiatives are successful in terms of confidence and language skills (Young 2018) • Fèisean nan Gàidheal tutors young people in traditional music, Gaelic song & drama. It aims to ‘bring our culture to our young people. ’ -Around 6, 000 young people take part in such tuition each year. -Research shows participation to be associated with positive attitudes to Gaelic and motivation to use the language (Broad & France 2005).

Cyd-destun mewnol/Internal context • Hyrwyddo iaith leiafrifol (gallu • Promoting a minority language (ability and especially use) ac yn enwedig defnydd) • Cysylltiad gyda hunaniaeth a • A connection with identity and diwylliant culture • Creu cymunedau ieithyddol • Creating language communities • Ond hefyd… – Creu dinasyddion ‘gwell’ – Gwella perfformiad academaidd – Lleihau anhafaliadau economaidd/NEETs • But also… – Creating ‘better’ citizens – Improving levels of academic achievement – Addressing socio-economic inequalities/NEETs

Cyd-destun mewnol/Internal context • Beth yw prif amcanion eich • mudiad? • I ba raddau rydych chi’n • asesu canlyniadau a thraweffaith eich gwaith? • Sut ydych yn gwneud hyn? • • Oes arfer dda? • • Beth yw’r heriau neu • rwystrau? • Oes cyfleon? • What are your organisation’s main aims? To what extent do you evaluate the outcomes and impact of your work? How do you do this? Is there good practice? What are the challenges or obstacles? Are there opportunities?

Cyd-destun mewnol/Internal context Capasiti (amser, sgiliau) ‘Pellter a deithiwyd’ Treigl amser Cymdeithas fel system cymhleth • Gofynion asiantaethau ariannu gwahanol • Heriau penodol ym maes iaith: • • – Defnydd iaith – Perthynas gyda ffactorau mwy haniaethol – Ai data meintiol sydd orau? Capacity (time, skills) ‘Distance travelled’ The passage of time Society as a complex system The demands of different funding agencies • Specific challenges in relation to language: • • • – Language use – Link to more abstract factors – Can quantitative data capture this?

Cyd-destun mewnol/Internal context • Cyfleon • Opportunities • Rydych chi’n adnabod eich ‘cleientiaid’ yn dda iawn • You know your ‘clients’ very • Mae gennych chi storfa fawr well o wybodaeth amdanynt • You possess a large store of • Mae cyfle i lunio agenda information about them gyffredin? • There is an opportunity to shape a common agenda?

Cyd-destun mewnol/Internal context • • Allbwn, canlyniad a thraweffaith • Allbwn: gweithgareddau, • prosesau, cynnyrch • Canlyniad: canlyniad y • gweithareddau a phrosesau • • Traweffaith: effaith ehangach a mwy hir dymor Output, outcome and impact Output: actions, processes, products Outcomes: the results of actions and processes Impact: broader and longer -term effects

Cyd-destun mewnol/Internal context • “While the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants’ lives on a broader scale” (Andrew Harding, LSE). • Path fath o dystiolaeth mae modd ei gasglu? /What kind of evidence is it possible to collect? • Pa fath o dystiolaeth a ddisgwylir i chi ei arddangos? /What kind of evidence are you expected to produce?

Cyd-destun mewnol/Internal context

Theori newid/Theory of change • Theori newid • Theory of change • Deiagram sy’n dangos • dylanwad rhaglen neu ymyrraeth ar gleientiaid • • Amlinellu yr holl weithgareddau a phrosesau (allbwn) • • Sut mae’r prosesau ayb yn arwain at ganlyniadau • • Natur traweffaith ehangach y canlyniadau A diagram that represents the effect of a programme or intervention on clients Outline all of the activities and processes (output) How these processes etc lead to outcomes The broader impact of the outcomes

Dull y NWA/The MSC method

Theori newid/Theory of change • “We believe that a clear, concise and convincing explanation of what you do, what impact you aim to have and how you believe you will have it is a vital foundation of any programme, and a prerequisite for effective evaluation” (NESTA, Guidance for Developing a Theory of Change for Your Programme).

Theori newid/Theory of change • Cymrwch 10 munud i lunio deiagram theori newid ar gyfer un o amcanion eich mudiad • Take 10 minutes to produce a theory of change diagram for one of your organization’s aims • A ydyw’n ddefnyddiol? • A ydyw’n arwain at gwestiynau/ystyriaethau newydd? • Is it useful? • Does it lead to new questions/considerations?

Camau nesaf/Next steps • Cymorth • Cyrff yng Nghymru a’r Alban yn barod i helpu – Prifysgol Aberystwyth – Prifysgol Caeredin – WISERD • Pa gefnogaeth sydd ei angen? • Help • Institutions in Scotland Wales are ready to help – Aberystwyth University – Edinburgh University – WISERD • What support is needed?

Camau nesaf/Next steps • Camau nesaf • Next steps • Ysgrifennu dogfen ar sail • Writing a document on y cyflwyniad, yr ymchwil the basis of the sy’n sail iddo a’r presentation, the mewnbwn gennych chi research that lies behind it and your input • Parhau i gydweithio? • Continuing to collaborate?