st Revised 21 CCLC External Evaluation Framework 2018

st Revised 21 CCLC External Evaluation Framework (2018 -2023) Terri Foulkes, Missouri After. School Network Wayne Mayfield, Ph. D, Office of Social and Economic Data Analysis

Guiding Principles • Grantees and sites should see and be able to use the data collected • Expectations (goals) should be consistent across all levels (site, grant, state) • Programs need support with interpreting and using their data • Evaluation framework should align with ESSA

Overall Purpose of the External Evaluation • To provide DESE with an detailed and accurate picture of the grantee and their progress on the statewide goals and objectives • To provide the grantee accurate and usable information for dissemination about their use of grant funds and to support their ability to make progress toward meeting and exceeding the statewide goals and objectives

Contributions to the Revisions • Meetings between MASN, OSEDA, and DESE • Dec 2017 – March 2018 • Ongoing consultation • Input from AREs • January 2018 • August 2018 • September 2018 • Grantee Focus Groups • March 2018 • June 2018

Goals and Objectives • Goals – Same • Objectives – Updated

2018 -2023 Goals and Objectives Goal 1 Objective 1. 1: Objective 1. 2: Support or increase student achievement and sense of competence in the areas of reading/language arts, mathematics, and science. The afterschool grantee will score satisfactory or above on the reading/communications arts rubric. The afterschool grantee will score satisfactory or above on the math rubric. The afterschool grantee will score satisfactory or above on the science rubric. Objective 1. 3: Goal 2 Objective 2. 1: Objective 2. 2: Objective 2. 3: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement. The afterschool site(s) will score satisfactory or above on the program quality improvement rubric. The afterschool site(s) will score satisfactory or above on the school day alignment rubric. The afterschool site(s) will score satisfactory or above on the broad array rubric. The afterschool grantee will score satisfactory or above on the family engagement rubric. Objective 2. 4: Goal 3 Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning. Objective 3. 1: The afterschool grantee will score satisfactory or above on the program attendance rubric. Objective 3. 2: The afterschool grantee will score satisfactory or above on the personal and social skills rubric. Objective 3. 3: The afterschool grantee will score satisfactory or above on the commitment to learning rubric.

The Influence of ESSA 1. Provide opportunities for academic enrichment, including providing tutorial services to help students, particularly students who attend low-performing schools, to meet the challenging State academic standards in at least math, reading, and science (subjects are Missouri specific); 2. offer students a broad array of additional services, programs, and activities; such as, youth development activities, service learning, nutrition and health education, drug and violence prevention programs, counseling programs, arts, music, physical fitness and wellness programs, technology education programs, financial literacy programs, environmental literacy programs, mathematics, science, career and technical programs, internship or apprenticeship programs, and other ties to an indemand industry sector or occupation for high school students that are designed to reinforce and complement the regular academic program of participating students; and 3. offer families of students served by community learning centers, opportunities for active and meaningful engagement in their children’s education, including opportunities for literacy and related educational development.

The Influence of ESSA Goal 1 Objective 2. 2 Academic Support Objective 2. 3 Goal 3 Broad Array Objective 2. 4 Family Engagement

Changing to the Rubric “… will score satisfactory or above on the … rubric. ” Incomplete Item A Item B Item C Less than Satisfactory More than Satisfactory Exemplary

Benefits of the Rubric • Capture the complexity of each objective Old Reading • 1. 1 – Y/N – 50% of youth will maintain or increase • 1. 4 – Y/N – 70% of youth will report a medium to high level of reading efficacy New Reading • 1. 1 – score satisfactory or above on the reading/communication arts rubric • Includes grades and efficacy

Benefits of the Rubric • Flexibility to have strengths and areas for improvement in same category • Recognizes programs that are doing more than satisfactory Incomplete Less than Satisfactory Item A More than Satisfactory Exemplary X Item B Item C Satisfactory X X

Shift from Site to Grantee • Goal 1, Objective 2. 4, and Goal 3 are calculated at the grantee level • This should significantly help grantees with multiple sites and age groups

Example – Reading Grades • Old – “At least 50% of youth per site maintain or increase their grade. ” Objective Status: Met or Not Met (at all sites) 1. 1 – Reading Not Met Grades If Not Met, which site(s) Data (for all sites) or missing data comments Site 5 – Not Met Site 1 – 100. 0 Site 2 – 86. 7 Site 3 – 82. 8 Site 4 – 100. 0 Site 5 – 49. 1 • New – “The afterschool grantee will score satisfactory or above on the reading/communication arts rubric. ” A Grades all 30+ 83. 9% Exemplary At least 70% of youth maintain or increase their grade in reading/language arts.

Scoring the Rubric • All rubrics are looking for “Satisfactory or Above” Incomplete Less than Satisfactory Item A Satisfactory More than Satisfactory X Item B X Item C X Incomplete Item A Less than Satisfactory More than Satisfactory Exemplary X Item B Item C Exemplary X X

Pass out Evaluation Rubric

Goal 1 - Academics Goal 1 Support or increase student achievement and sense of competence in the areas of reading/language arts, mathematics, and science. Objective 1. 1: The afterschool grantee will score satisfactory or above on the reading/communications arts rubric. A B Grades - all 57% 30+ Grades - 65% students with pre-grade in "need" category MAP 55% Satisfactory End of Course NA (EOC) Exams - English II Efficacy 82% More than Satisfactory C D E More than Satisfactory At least 50% of youth maintain or increase their grade in reading/language arts. At least 60% of students identified as "need" increase their grade. At least 50% of the students with reading/language arts MAP scores of Proficient or above OR at least a 5% increase in the percentage of students scoring Proficient or above from the previous year. At least 55% of the students with reading/language arts EOC scores of Proficient or above. At least 80% of youth per site will report a medium to high level of reading efficacy as measured by items on the Youth Survey (average score of 3. 5 or higher).

Goal 3 – Youth Outcomes Goal 3 Objective 3. 1: A B C D E F G Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning. The afterschool grantee will score satisfactory or above on the program attendance rubric. (Elem = K-5, MS/JR = 6 -8, HS = 9 -12. ) Less than Satisfactory More than Satisfactory Exemplary Actual 30+ day attendance is less attendance is at least than 80% of proposed attendance. Less than 50% of At least 50% of youth Elementary – youth attend 30+ days At least 50% of youth Elementary stretch - Less than 50% of youth attend 60+ days At least 40% of youth Middle School/Junior Less than 40% of youth attend 30+ days. High - 30+ days At least 50% of youth Middle School /Junior Less than 50% of youth attend 45+ days. High stretch - 45+ days Less than 30% of At least 30% of youth High School – youth attend 30+ days At least 50% of youth High School stretch - Less than 50% of youth attend 30+ days Proposed vs Actual Attendance Actual 30+ day attendance is at least 90% of proposed attendance. At least 60% of youth attend 30+ days. At least 60% of youth attend 60+ days. At least 50% of youth attend 30+ days. At least 60% of youth attend 45+ days. Actual 30+ day attendance is at least 100% of proposed attendance. At least 70% of youth attend 30+ days. At least 70% of youth attend 60+ days. At least 60% of youth attend 30+ days. At least 70% of youth attend 45+ days. At least 40% of youth attend 30+ days. At least 60% of youth attend 30+ days. At least 50% of youth attend 30+ days. At least 70% of youth attend 30+ days.

Goal 3 – Youth Outcomes Objective 3. 2: The afterschool grantee will score satisfactory or above on the Personal and Social Skills rubric. Personal and No youth Social Skills scale surveys. A B Objective 3. 3: School Day Discipline (DESE Data) At least 80% of youth At least 70% of youth indicate a medium to high level of personal and social skills as measured by items on on the Youth Survey. Less than 70% of youth have At least 80% of youth no in-building or out-ofhave no in-building or school suspensions. out-of-school suspensions. At least 90% of youth indicate a medium to high level of personal and social skills as measured by items on the Youth Survey. At least 90% of youth have no in-building or out-ofschool suspensions. The afterschool grantee will score satisfactory or above on the Personal and Social Skills rubric. Commitment to No youth Learning scale surveys. A School Day Attendance B Less than 70% of youth indicate a medium to high level of personal and social skills as measured by items on the Youth Survey. Less than 70% of youth At least 80% of youth At least 70% of youth At least 90% of youth indicate a medium to high indicate a medium to level of commitment to high level of learning as measured by commitment to learning items on the Youth Survey. learning as measured by items on the Youth Survey. Less than 60% of youth At least 50% of youth At least 60% of youth At least 70% of youth meet or exceed the school meet or exceed the district's average rate of school district’s average school district's average school day attendance. rate of school day average rate of attendance.

Goal 2 – Program Quality • Objectives 2. 1 – Observed Program Quality, 2. 2 – School Day Alignment, and 2. 3 – Broad Array are at the site level • All sites will need to score “Satisfactory or above” • Objective 2. 4 – Family Engagement is at the grantee level

Goal 2. 1 – The afterschool site(s) will score satisfactory or above on the program quality improvement rubric. • Program Quality Assessment (PQA) • 2. 9 or above on PQA tool OR if less than 2. 9 at least 50% of site staff attend an extra YWM training • Staff Program Quality scale • Youth Program Quality scale • Family Program Quality Scale

The Influence of ESSA • (1) COMMUNITY LEARNING CENTER. -The term "community learning center'' means an entity that(A) assists students to meet the challenging State academic standards by providing the students with academic enrichment activities and a broad array of other activities … (i) reinforce and complement the regular academic programs of the schools attended by the students served; and (ii) are targeted to the students' academic needs and aligned with the instruction students receive during the school day; and

The Influence of ESSA Targeted Strategies Communication Plans State Standards Academic Support Schedules Lesson Plans Broad Array Family Engagement

Goal 2. 2 – The afterschool site(s) will score satisfactory or above on the school day alignment rubric. • Consistency of Curriculum and Coordination of Content • Documenting State Standards in Lesson Plans • School Day Administrator Surveys • Staff/Coordinator School Day Linkages Scale

Goal 2. 2. A – Consistency of Curriculum and Coordination of Content • Consistency of Curriculum and Coordination of Content • The plan should include details about how school day staff and afterschool staff will ensure consistency of curriculum and coordination (but not replication) of content, including • information about what content the afterschool program will cover that is consistent/coordinated, but not replicated, • the type of individual student information that will be shared to help determine targeted interventions, and • methods of ongoing communication between school and afterschool team members such as • what will be communicated about students • methods of communication • regularity/timing of communication

Goal 2. 3 – The afterschool site will score satisfactory or above on the broad array rubric. • Targeted Academic Strategies • Social and Emotional Learning • Lesson Plans • Weekly Schedule - Exposure to a Variety of Topics • Daily Schedule - Variety of Activities Each Day • Choice of Activities

Goal 2. 3. A – Targeted Academic Strategies • A targeted academic strategy is an ongoing, intentional intervention selected and implemented to meet the identified academic needs of designated students. The "Targeted Academic Strategies Summary" document should be completed for each targeted academic strategy.

Life Skills Education Academic Support Homework Help Tutoring Academic Enrichment Targeted Academic Strategy Health, Recreation, Fitness Broad Array Service Learning College and Career Readiness STEM Fine and Performing Arts SEL Snack/Meals Free Time (indoor or outdoor)

The Influence of ESSA • (1) COMMUNITY LEARNING CENTER. -The term "community learning center'' means an entity that(B) offers families of students served by such center opportunities for active and meaningful engagement in their children's education, including opportunities for literacy and related educational development.

The Influence of ESSA Family & Child Enrichment Educational Development for Adult Family Members Academic Support Broad Array Family Engagement

Goal 2. 4 – The afterschool grantee will score satisfactory or above on the family engagement rubric. • Family & Child Academic Enrichment Opportunities • Educational Development for Adult Family Members of Students Served • Family Strengthening Families Scale • Staff/Coordinator Strengthening Families Scale

Questions

Life Skills Education Academic Support Homework Help Tutoring Health, Recreation, Fitness Broad Array Service Learning Academic Enrichment College and Career Readiness STEM Fine and Performing Arts SEL Snack/Meals Free Time (indoor or outdoor)

Changes to the Process Ev al ua ti on D es ig • MASN n • OSEDA • DESE D at a Co lle cti • MASN on • Grantees • External Evaluator D at a Re po rt G en • OSEDA er • Weikart ati on • G ui de d Re fle cti on Re D vi oc e u External Evaluator w m • External of en D Evaluator t at • Program a Director Gr an te e Le ve l Ac ti • Grantees on • Coaches Pl an s

Grantee Level Cycle of Improvement Improve Implement Plan Guided Reflection • Training, resources, etc. • Revisit the Action Plan at 2 nd visit and at other contact points • Local Context • Data Review • Written responses • Includes a review of previous year’s goal Program Level Action Plan • Developed by Program Director and ARE • Based on External Evaluators recommendation • Submitted to Brad for data entry • Copy provided to External Evaluator Plan Assess

Comprehensive Data Collection and Improvement Framework

Apples-to-Apples Evaluation System Additional Analysis Ev al u at io n D es • MASN ig • OSEDA n • DESE D at a C ol le • MASN ct • Grantees io n D at a R e p or t G e • OSEDA n • Weikart er at io n G • OSEDA ui • Weikart d e d R ef le ct io n D o • Local Evaluator c u m e Planning nt with Data • Sites • Coaches Statewide Evaluation • OSEDA • Weikart G ra nt e e Le v el A ct • Grantees io • Coaches n Pl a Site Level ns Action Plans • Sites • Coaches State Grant Site

Use of Evaluation Results to Develop and Refine the Afterschool Program FY 17 Guided Reflection Recommendations FY 18 Guided Reflection Review 2017 -18 Quality Action Plan