Big Brothers Big Sisters Saskatchewan An evaluation proposal

Big Brothers Big Sisters Saskatchewan An evaluation proposal June 15, 2020

ACKNOWLEDGEMENT We learn and work on the traditional territory of the Neutral, Anishinaabeg, and Haudenosaunee peoples. The University of Waterloo is situated on the Haldimand Tract, land promised to the Six Nations that includes six miles on each side of the Grand River. Our words are insufficient. We must take action in all our evaluation work towards reconciliation in Canada.

MEET THE TEAM Isabella Romano Amanda Doggett Claire Buchan Alle Butler Evaluation Design Methods Stakeholder Engagement Project Management

OUTLINE ● Program overview ● Stakeholder engagement ● Logic model ● Evaluation design ● Data collection methods ● Evaluation matrix ● Knowledge translation ● Challenges & mitigations ● Schedule of activities ● Credentialed Evaluator competencies

Program Overview

PROGRAM OVERVIEW Big Brothers Big Sisters of Canada is a Federation comprised of 135 member agencies servicing more than 1000 communities across the country Mission: to enable life-changing mentoring relationships to ignite the power and potential of young people Big Brothers Big Sisters Saskatchewan: Organization consisting of 5 agencies across the province (BBBS Regina and Area; YWCA Big Sisters Regina; BBBS Saskatoon and Area; BBBS Yorkton; BBBS Lloydminster) Developmental Relationships: Group & Individual Mentoring programs • Big Siblings One-on-one Mentoring • In-School Mentoring • Shared Journey Program

PROGRAM OVERVIEW One-on-one Mentoring: Youth in the community are matched with a volunteer mentor • Meet once per week • Model positive behaviours or lifestyles for youth • Engage in activities within the community • 1 year commitment In-School Mentoring: Youth in elementary school matched with a volunteer mentor • 1 hour a week • Engage in activities together • Board games • Crafts • Sports • 1 school year Shared Journey: Indigenous and Non-Indigenous mentors and youth work together, learn together and mentor each other • 1 x per week • Identify actionable items to mobilize reconciliation within their own lives • 10 week program

POPULATION OVERVIEW Big Brothers Big Sisters has a vested interest in improving the lives and outcomes of young people across the province, including those from Indigenous, newcomer, racialized, and LGBTQ 2 S+ communities BBBS Saskatchewan serves: • 13 communities (60% urban; 40% rural) • 1, 000 young people (5 -20 s) • >50% youth severed are First Nations or Métis • 5% are newcomers BBBS youth community: • ~50% youth live in conditions that could negatively impact their development and well-being • Predominantly from low income families / single-parent families • 25% involved in social services • 25% report mental health concerns

Stakeholder Engagement

STAKEHOLDERS Internal Stakeholders: Individuals or groups directly involved in BBBG Saskatchewan External Stakeholders: Individuals or groups not directly within BBBG Saskatchewan but who are invested in its performance

STAKEHOLDER ENGAGEMENT PLAN Advisory Committee • • Youth Advisory Committee To help guide and shape our evaluation plan To advise on feasibility, appropriateness, utility To provide contextual/cultural lens To help make sense of our findings

Logic Model

LOGIC MODEL NARRATIVE Outline the program objective, ongoing activities, outputs and outcomes of the program ● ● ● Primary audience: Youth (mentees) Secondary audiences: mentors Project objective, activities, inputs and outputs were identified Assumptions (4) External factors (3) Potential risks (1)

Program: BBBS seeks to enable life-changing mentioning relationships to ignite the full potential of young people Project Goal: to demonstrate a collective story about the BBBS program work and alignment with BBBS agencies across Canada Inputs Situation One in four children live in poverty in Saskatchewan Staff Mentors Funding Mental illness is a prominent challenge within Saskatchewan Approximately half of youth that are involved in BBBS live in adverse environments BBBS programs across Canada have previously shown that mentorship has had a positive impact on youth and mentors Time Materials and supplies (permitting virtual meetings and events) Outputs Activities One-on-One Mentoring: Youth in the community are matched with a volunteer mentor Meet once a week One year commitment Meeting site: local community Model positive behaviours or lifestyles for youth Participate in activities together within their community In-School Mentoring: Youth are matched with a volunteer mentor Meet once a week throughout school year Meeting site: school Participate in activities together at school to support self-esteem building Shared Journey: Indigenous and non-Indigenous youth and mentors meet at site (i. e. , school) Meet once a week for 10 weeks Participate in weekly modules RE: reconciliation, diversity, & history Identify actionable items to mobilize reconciliation within their own lives For all activities: - # of mentees - # of screened & trained mentors - # of maintained relationship One-on-One Mentoring: - Type of activities engaged in - # of hours engaged per week - # of parents/guardians engaged - Strength of bond - Satisfaction with developmental relationship - # of successful mentee/mentor matches In-School Mentoring: - Rate of student graduation - School attendance - # of weeks mentee & mentors meet Shared Journey: # of students referred to program % of students that thought program facilitated positive change % of students with improved understanding of reconciliation # of students willing to act on new knowledge Outcomes - - Shorter Term (<1 year) Boosts self-esteem and wellbeing Develop more connections with others Mentees develop strong bond with mentor ↑ understanding of reconciliation Medium Term (1 -3 years) Broaden worldviews ↑ sense of self-fulfilment ↑ sense of connection with community ↑ communication and social skills Maintenance of mentee/mentor relationship beyond commitment period Longer Term (>3 years) Social emotional competence Self-management and -awareness Responsible decision making ↑ social skills with adults Mental Health & Wellbeing Positive self identity Mental wellness ↑ sense of connection with community, social inclusion & empowerment Educational engagement & employment readiness School connectedness & commitment to learning ↑ graduation rates Assumptions: 1) Youth are willing to meet mentors and participate in activities, 2) Mentors model positive behaviours, 3) Students attend school, 4) Consent is given by schools or parents/guardians Risks: 1) The program might not be accessible to all participants External Factors: 1) Changes in funding priorities, 2) Cultural norms, 3) Judgement from peers or guardians

PROGRAM: BBBS seeks to enable life-changing mentioning relationships to ignite the full potential of young people PROJECT GOAL: to demonstrate a collective story about the BBBS program work and alignment with BBBS agencies across Canada SITUATION One in four children live in poverty in Saskatchewan Mental illness is a prominent challenge within Saskatchewan Approximately half of youth that are involved in BBBS live in adverse environments BBBS programs across Canada have previously shown that mentorship has had a positive impact on youth and mentors

INPUTS Staff Mentors Funding Time Materials and supplies (permitting virtual meetings and events) ACTIVITIES One-on-One Mentoring: Youth in the community are matched with a volunteer mentor - Meet once a week - One year commitment - Meeting site: local community - Model positive behaviours or lifestyles for youth - Participate in activities together within their community In-School Mentoring: Youth are matched with a volunteer mentor - Meet once a week throughout school year - Meeting site: school - Participate in activities together at school to support selfesteem building Shared Journey: Indigenous and non-Indigenous youth and mentors meet at site (i. e. , school) - Meet once a week for 10 weeks - Participate in modules RE: reconciliation, diversity, & history - Identify actionable items to mobilize reconciliation within their own lives

OUTPUTS For all activities: - # of mentees - # of screened & trained mentors - # of maintained relationship One-on-One Mentoring: - Type of activities engaged in - # of hours engaged per week - # of parents/guardians engaged - Strength of bond - Satisfaction with developmental relationship - # of successful mentee/mentor matches In-School Mentoring: - Rate of student graduation - School attendance - # of weeks mentee & mentors meet Shared Journey: - # of students referred to program - % of students that thought program facilitated positive change - % of students with improved understanding of reconciliation - # of students willing to act on new knowledge OUTCOMES - Shorter Term (<1 year) Boosts self-esteem and wellbeing Develop more connections with others Mentees develop strong bond with mentor ↑ understanding of reconciliation - Medium Term (1 -3 years) Broaden worldviews ↑ sense of self-fulfilment ↑ sense of connection with community ↑ communication and social skills Maintenance of mentee/mentor relationship Longer Term (>3 years) Social emotional competence - Self-management and -awareness - Responsible decision making - ↑ social skills with adults Mental Health & Wellbeing - Positive self identity - Mental wellness - ↑ sense of connection with community, social inclusion & empowerment Educational engagement & employment readiness - School connectedness & commitment to learning - ↑ graduation rates

ASSUMPTIONS, RISKS, AND EXTERNAL FACTORS Logic Model Assumptions: 1) Youth are willing to meet mentors and participate in activities, 2) Mentors model positive behaviours, 3) Students attend school 4) Consent is given by schools or parents/guardians Risks: 1) The program might not be accessible to all participants External Factors: 1) Changes in funding priorities, 2) Cultural norms, 3) Judgement from peers or guardians

Evaluation Design

Goals EVALUATION TYPE - Impact Determine the impact of: • One-on-One Mentoring • In-School Mentoring • Shared Journey Program Determine the impact on: • Mentees • Mentors Tell a collective story about the work of of the 5 BBBS Saskatchewan agencies, and demonstrate alignment with other BBBS agencies

EVALUATION APPROACH Participatory Why? Culturally Responsive Evaluation Framework The lived experience of youth participating in BBBS programs, as well as the experiences of those close to the program, are essential to this evaluation. Their feedback should be the guiding force behind decisionmaking. How? • Two Advisory Committees: Advisory Committee Youth Advisory Committee • Involvement throughout the evaluation • Advisory meetings • Participatory data analysis • Knowledge translation (Adapted from Frierson et. al. , 2002; Hood, S. et al. , 2015)

EVALUATION SCOPE Inclusive of… • All 5 BBBS Saskatchewan agencies • BBBS Regina and Area • YWCA Big Sisters Regina • BBBS Saskatoon and Area • BBBS Yorkton • BBBS Lloydminster • All three developmental relationships programs • Big Brothers/Big Sisters/Big Siblings One-on-One Mentoring • In-School Mentoring • Shared Journey

Data Collection Methods

Virtual Focus Groups Document Review Art Voice Questionnaires Participatory Data Analysis

DOCUMENT REVIEW of existing documents What? • Review existing intake assessments and program feedback to triangulate data from other data collection approaches Why? • • • Understand currently available information and indicators Provide indicators to inform the impact of the program, as well as efficacy of these intake and feedback tools Inform some of the question prompts for VFGs

VIRTUAL FOCUS GROUPS - Mentors What? Why? One VFG for each program (One-on-One Mentoring, In-School Mentoring, Shared Journey) - 3 in total This method aims to identify and explain mentors perception of: • • • Function similar to online discussion board - asynchronous Participants provided with URL and login 7 -day access period Recruitment: • Email Incentive: 2 draws for $50 entertainment gift card per VFGs Aiming for n=3 -6 from each site ● ● Perceived benefits of the three BBBS programs to the mentees ○ Social skills ○ Confidence in abilities ○ Self-efficacy ○ Technical skills ○ Knowledge of reconciliation, history of Canada, First Nations, Metis, and Settler communities and culture Perceived benefits of the three BBBS programs to the mentors themselves ○ Current and future benefits of being a mentor

VIRTUAL FOCUS GROUPS - Mentees aged >12 years What? Why? One VFG for each program (One-on-One Mentoring, In-School Mentoring, Shared Journey) - 3 in total This method aims to identify and explain mentors perception of: • • • Function similar to online discussion board - asynchronous Participants provided with URL and login 7 -day access period Recruitment: • Through mentors, parents Incentive: 2 draws for $50 entertainment gift card per VFGs Aiming for n=3 -6 from each site ● Perceived benefits of the three BBBS programs to the mentees ○ Social skills ○ Confidence in abilities ○ Self-efficacy ○ Technical skills ○ Knowledge of reconciliation, history of Canada, First Nations, Metis, and Settler communities and culture

QUESTIONNAIRES - Referral parties & parents What? • • Anonymous online questionnaire distributed to referral parties (school staff, counsellors, social services) and parents Mix of quantitative and qualitative indicators to gauge general perspective of program impact on mentees • Social skills • Wellbeing • Knowledge of reconciliation, history of Canada, First Nations, Metis, and Settler communities and culture Incentive for parental participation: draw for 2 $50 grocery gift cards Aim for n=50, n=30 referral parties Why? • • Many referral parties are invested in the success of the program Individual who refer students likely have referred more than one individual and may have perspective to summarize impact • Will develop questions and gauge utility during initial advisory group meeting Referral parties may engage with youth on a frequent basis Parents have unique perspectives on their children’s development

ART VOICE - Mentees aged >5 years What? • • • An enhancement of Photo Voice Participants asked to represent their experience(s) with BBBS through: • Taking photo(s) • Creating an art piece (any medium) Associated narrative can be written or audio-recorded Evaluation team will work with BBBS agencies if participants do not have access to materials (camera, art supplies) Incentive: 2 draws for $25 entertainment gift card per site, Art Gallery recognition Aiming for n=10 for each location Why? • • • Allows for the capture of more abstract concepts and experiences (Wang, 2006) Allows participants to express themselves free of judgement from others, and is less invasive Art is a more natural medium for younger children (Dreissnack & Furukawa, 2012) Can benefit participants through empowerment and changes in selfperception (Budgid et al, 2018) Valuable approach for knowledge translation

PARTICIPATORY DATA ANALYSIS What? • • • Sense-making workshop to interpret and synthesize the data across methods Think-pair-share style collaborative activity Virtual to allow for people from all different sites to be involved Advisory Committee: • Record review • VFGs with mentors • Questionnaires Youth Advisory Committee: • VFGs with mentees • Art Voice Why? • • Aligns with the participatory approach, as stakeholders hold unique insight through program involvement and lived experiences Helps to inform how knowledge can be best translated to different stakeholders

Evaluation Matrix

KEY EVALUATION QUESTIONS Across all BBBS Saskatchewan agencies: 1. What is the impact of One-on-One Mentoring? 2. What is the impact of In-School Mentoring? 3. What is the impact of the Shared Journey program?

EVALUATION MATRIX Sub-questions 1. What is the impact of One-on-One Mentoring? Indicators What are experiences with One-on-One Mentoring like? • • Type of activities engaged in # of hours engaged per week Frequency of One-on-One meetings Reported satisfaction 1. 2 Do mentees and mentors benefit from One-on-One Mentoring? If so, to what extent? • • • Perceived mentorship bond quality Changes in self-esteem and wellbeing Perceived connectedness with others Mentor satisfaction Reported drawbacks, if any 1. 3 How does this program align with other BBBS agencies? • • Perceived alignment Perceived areas for improvement # of successful matches Average duration of mentorship 1. 1 Data source(s) Method • Mentees • Mentors • Program records • VFGs • Art Voice • Record review • • Mentees Mentors Program records Referral parties & parents • Advisory committees • • • Mentors • Program records • (Non-youth) advisory committee only • VFGs • Record review • Participatory analyses VFGs Art Voice Record review Questionnaire Participatory analyses

EVALUATION MATRIX Sub-questions 2. 1 What are experiences with In-School Mentoring like? 2. What is the impact of In-School Mentoring? Indicators • Type of activities engaged in • # of weeks mentees & mentors meet • Reported satisfaction 2. 2 Do mentees and mentors benefit from In-School Mentoring? If so, to what extent? • • Perceived mentorship bond quality Changes in self-esteem and wellbeing Changes in social skills Changes in technical skills Perceived desire to stay in school Rates of truancy, graduation Mentor satisfaction Reported drawbacks, if any 2. 3 How does this program align with other BBBS agencies? • • Perceived alignment Perceived areas for improvement # of participants per week Participant attrition Data source(s) Method • Mentees • Mentors • Program records • VFGs • Art Voice • Record review • • Mentees Mentors Program records Referral parties & parents • Advisory committees • • • Mentors • Program records • (Non-youth) advisory committee only • VFGs • Record review • Participatory analyses VFGs Art Voice Record review Questionnaire Participatory analyses

EVALUATION MATRIX Sub-questions 3. 1 What are experiences in the Shared Journey program like? 3. What is the impact of the Shared Journey program? Indicators • • • # of participants Perceived quality of engagement % completion per module Reported satisfaction Perceived engagement and reflection Data source(s) • Mentees • Mentors • Program records • VFGs • Art Voice • Record review • • 3. 2 Do mentees and mentors benefit from One-on-One Mentoring? If so, to what extent? • Changes in understanding of: reconciliation, history of Canada, knowledge of First Nations, Métis, and Settler communities and cultures • Mentor satisfaction • Reported drawbacks, if any • • 3. 3 How does this program align with other BBBS agencies? • • • Mentors • Program records • (Non-youth) advisory committee only Perceived alignment Perceived areas for improvement # of students referred to program Participant attrition Method Mentees Mentors Program records Advisory committees VFGs Art Voice Record review Participatory analyses • VFGs • Record review • Participatory analyses

Knowledge Translation

KNOWLEDGE TRANSLATION STRATEGIES We believe that knowledge translation (KT) is an essential component of any evaluation. While much of the KT strategies will be informed by the participatory approach, in particular during the participatory sense-making data analysis stage, we currently propose a few potential strategies: • 1 -3 -25 report • • 1 -page summary will be provided in infographic format Webinar Art Gallery Appreciation Night: • Showcase of art voice pieces and accompanying narratives (option for pieces to be credited or anonymous) • Invite all stakeholders to join • Food and beverages served as a thank you • One gallery in each location for accessibility

Challenges & Mitigations

CHALLENGES & MITIGATIONS Challenge 1. Cultural barriers 2. Recruitment of mentees, mentors, and schools 3. Limited time and budget provided Mitigation strategy • • Advisory committees and participatory analysis to interpret meaning of results Advisory committees may be able to express cultural elements Cultural training for evaluators prior to evaluation • Through the use of virtual focus groups: • Option to remain anonymous • Transportation not required • Mentors and mentees can participate at a time that best suits them • Art-based participation provides a more engaging platform for participation • Consent will be given from schools, school boards and parent/guardians • All data collect and most knowledge dissemination activities are conducted virtually

Schedule of Activities

TIMELINE Week Methods and Instrument Development Advisory Committee Meetings Recruitment Data Collection Report Generation and Analysis Knowledge Translation 1 2 3 4 5 6 7 8 9 10

Credentialed Evaluator Competencies

CREDENTIALED EVALUATOR COMPETENCIES Orbis Solutions strives to conduct the proposed evaluation in accordance with Credentialed Evaluator competencies: 4. 6 Communicates project progress to all concerned • Advisory Committee representing stakeholders • Communication of progress and Knowledge Translation to all audiences 5. 1 Uses communication strategies appropriate to the cultural, linguistic, social, and political context • Participatory evaluation approach & data analyses • Gathering data through use of Art Voice, Virtual Focus Groups (Canadian Evaluation Society, 2018)

Thank you! E-mail: orbis. evaluations@gmail. com Acknowledgements ● ● ● Drs. Skinner and Yessis CES Case Competition Staff & Funders Audience members

REFERENCES Budig, K. , Diez, J. , Conde, P. , Sastre, M. , Hernán, M. , & Franco, M. (2018). Photovoice and empowerment: evaluating the transformative potential of a participatory action research project. BMC public health , 18(1), 432. Canadian Evaluation Society [CES]. (2018). Competencies for Canadian Evaluation Practice, 2018 Edition. https: //evaluationcanada. ca/txt/2_competencies_cdn_evaluation_practice. pdf Driessnack, M. , & Furukawa, R. (2012). Arts‐based data collection techniques used in child research. Journal for Specialists in Pediatric Nursing , 17(1), 3 -9. Frierson, H. T. , Hood, S. , & Hughes, G. B. (2002). Strategies that address culturally responsive evaluation. In J. Frechtling (ed. ). , The 2002 user friendly handbook for project evaluation (pp. 63 -73). Arlington, VA: National Science Foundation. Hood, S. , Hopson, R. K. , & Kirkhart, K. (2015). Culturally responsive evaluation. In K. E. Newcomer, H. P. Hatry & J. S. Wholey (eds. ), Handbook of Practical Program Evaluation (4 th Edition) (pp. 281 -317). Hoboken, NJ: John Wiley & Sons. Wang, C. C. (2006). Youth participation in photovoice as a strategy for community change. Journal of community practice 147 -161. Images: 1. All illustrations were obtained from: https: //www. freepik. com/free-photos-vectors , 14(1 -2),