Measures of Student Learning CFN 204 Summer 2013

Measures of Student Learning CFN 204 Summer 2013 CFN 204 · Diane Foley · Network Leader 1

Purpose & Context This work, particularly around Mo. SL, is very nuanced and complex. There are places where some policy decisions are still being made. Throughout the day we want you to capture your reflections and thinking to help you process and distinguish between what will become feedback to the DOE and what you need to move the work forward in your schools. The goal for today is to understand, so that you are ready to lead the work in your schools. The day is packed with content, but we will be making connections throughout the day and the institute so that you can see how to pull it all together for your schools. CFN 204 · Diane Foley · Network Leader 2

DOE Resources and Training The following resources will assist principals and the School Local Measures Committee throughout the decision-making process and implementation: • Measures of Student Learning Guide: You can find more information on assessment and measurement options, as well as sample selections in the Guide, found at http: //www. learndoe. org/dhr/recording-mosl-overview/. • Interactive Selection Tool: The DOE will release a tool to select assessments and measurements in August. • MOSL Specialists: Experts to help networks and schools implement in the fall. Additional training opportunities: • MOSL Specialists will be providing additional support to networks, who will in turn support schools, as the school year begins. CFN 204 · Diane Foley · Network Leader 3

Mo. SL Options Template Required assessments are checked and shaded in gray. State Measures decisions made by principal are color-coded in orange and Local Measures decisions made by the school committee are color-coded in blue (same as Mo. SL guides). For State Measures, teachers who teach both ELA & math must have an ELA and math component for their State Measure, but not their Local Measure. The Local Measure can be any subject they teach. For schools that departmentalize, teachers are only responsible for the content they teach. Pending policy decisions, for Local Measures, school committees may select up to 2 measures per group of similar teachers, i. e. same subject and grade. (2 measures = 1 assessment with 2 populations, or 2 assessments) For example: 1 assessment, 2 populations: ELA state exam – school and school lowest third 2 assessments: ELA state exam – school, ELA performance tasks – individual CFN 204 · Diane Foley · Network Leader 4

Mo. SL: Measures of Student Learning (40%) All teachers will receive: §Two different Measures of Student Learning (40%) § State Measures § Local Measures §Initial planning conference and summative end of year conference to include discussion of student outcomes (Will be covered on Aug. 8 th) §Multiple measures provide a more valid, robust picture of teacher performance, providing teachers with multiple sources of feedback CFN 204 · Diane Foley · Network Leader 5

State and Local Measures Although principals make decisions about State Measures and the School Local Measures Committee makes decisions about Local Measures, these two decisions go hand in hand. There are many reasons why it is important for the two measures to be determined in collaboration: 1. Teachers will receive the best picture of student growth with multiple measures, each telling a slightly different story of student performance. For example: Principal reinforces a focus on school-wide learning by selecting a school-wide measure on a State Assessment for a given group of teachers’ State Measure. School Local Measures Committee then decides to base Local Measure on individual teacher performance on a 3 rd party assessment (where possible) to include a variety of assessment perspectives. 2. Per law, teachers may not have the same exact measure for both State and Local (e. g. , 4 th grade teacher cannot have an individual classroom growth measure on ELA/Math state tests count for 40% of overall rating. Her Local Measure would have to measure another population either the lowest 1/3 or group measures such as school or grade). 3. The more alike the State and Local Measures are to each other, the more likely a teacher will receive the same rating on each. This may have positive or negative consequences. State Measures Local Measures CFN 204 · Diane Foley · Network Leader 6

Long-term Vision MORE OPTIONS More assessment options each year LEARN WITH EXPERIENCE Schools reselect options each year, learning with experience CFN 204 · Diane Foley · Network Leader 7

Terminology – Common Mix Ups State Measures (20%) The MOSL category that includes State Assessments or, where there are no State Assessments, a list of allowable assessments that can be used (chosen by principal where there is choice). Local Measures (20%) The MOSL category with options chosen by the School Local Measures Committee and submitted to the principal who may accept the recommendation or opt for the default measures. State assessments can be used for both state and local measures. When we are talking about state tests, it doesn’t always mean we are talking about state measures. The state made decisions about requirements on both the state and local side. Principals have a role in both as well. Local assessments – NYC performance assessments – are required for different grades on both the state and local side. Therefore, when we talk about local assessments it does not always mean we are talking about local measures. CFN 204 · Diane Foley · Network Leader 8

Local State Individual Class ELA Lowest 1/3, Grade, School State ELA Grade 4/5 CFN 204 · Diane Foley · Network Leader 9

Terminology – Common Mix Ups For each measure, the following three items need to be considered: Student growth is measured based on one or more of the following types of assessments: State Assessments, 3 rd Party Assessments, or NYC Performance Assessments. Assessment(s) Target Population Refers to the students included in the measure – this can include students school-wide, in a particular grade level, or only those students a teacher teaches. If the same assessment is selected for the State and Local Measures, the target population for the Local Measures will be the lowest third of students. Growth Measurements The method by which student growth is measured on a given assessment. For MOSL, options include teacher/principal goalsetting or growth models. Even though there are only 3 main decisions that need to be made for each measure, decisions impact each other and all options are not available in all cases. CFN 204 · Diane Foley · Network Leader 10

NYC Assessment Options Information on assessments available for specific grades/subjects is available in the Measures of Student Learning Guides (http: //www. learndoe. org/dhr/recording-mosl-overview/). The following parameters guided the selection of assessments available for each grade and subject: • State rules regarding allowable assessments: SED released rules about what assessments are allowed in each grade and subject, including specific rules about 3 rd Party and NYC Performance Assessments. • Assessment quality: NYC assessment options were evaluated to ensure instructional value, validity and reliability. • Security and comparability: Assessments must be available across all NYC schools and students cannot have access to them prior to administration. School-created assessments are not allowed for outcome assessments. * • Existing practice in schools: Whenever possible, NYC assessment options build upon existing assessment practices in NYC schools. *Where baseline selections need to be made, schools may use school-based/created assessments as baselines. Schools may NOT use school-based/created assessments for final outcomes. CFN 204 · Diane Foley · Network Leader 11

NYC Assessment Options NYC Assessment Type* Target Population Options** Description Example (K-5) State Assessments Individual Grade School Measure the performance of students based on state-created assessments. 3 -8 ELA & Math 4 & 8 Science NYSESLAT MS Regents: Integrated Algebra, Living Environment, Earth Science 3 rd Party Assessments Individual School Are created by assessment experts. Assessment format varies (multiple choice, performance based, etc. ) Math: K-2 Discovery Math, 3 -8 Scantron ELA: 3 rd grade Scantron, 3 rd grade Degrees of Reading Power (DRP) NYC Performance Assessments (developed w/ NYC Teachers) Individual Authentic tasks (e. g. , evidencebased essays), scored against a common rubric. Created by DOE, NYC teachers, curriculum and assessment experts to be used as Measures of Student Learning in teacher evaluation. K-8 ELA (2 options: tasks & running records) rd 3 grade Math 4, 6 & 7 Science 6 -8 Social Studies Integrated Algebra Living Environment Only state-approved 3 rd Party Assessments can be used. Not all NYC assessments are on this list. A final list will be approved on August 1. **NYC is not using all 3 rd Party Assessments that were previously approved for use throughout the state. ** NYCDOE is funding 3 rd Party Assessments used for teacher evaluation. 3 rd Party & NYC Performance Assessments: §Include both pre- and post-tests. §Teachers (per law) can score their own baselines/pre-tests, but CANNOT score their own post-test results. CFN 204 · Diane Foley · Network Leader 12

Activity: Preview NYC Performance Task Review the NYC Performance Task and aligned Rubric sample. Use the template to guide your reflections and thinking by considering the following prompts: 3 key points you learned from the NYC Performance Task and Rubric. 2 questions or implications of the assessment. 1 message to present to school/local measures committee. CFN 204 · Diane Foley · Network Leader 13

Baseline Assessments Assessment Baseline Recommendation Other Baseline Options 3 rd Grade Math/ELA NYC Performance Assessments Scantron ELA & Math Degrees of Reading Power (DRP) ELA School based ESL Prior Year NYSESLAT LAB-R (where prior year NYSESLAT isn’t available) School based 4 th Grade Science 3 rd Grade Math, NYC Science Performance Assessment School Based 8 th Grade Science 7 th Grade Math, NYC Living Environment Performance Assessment School Based A baseline assessment is one given in the beginning of the year to assess student starting points. NYC Performance Assessments and approved 3 rd Party Assessments come with baseline (pre) assessments. The principal must select a baseline assessment when none is included or where there is a State Assessment without historical data (e. g. , 3 rd grade). There are only a few instances where baselines will need to be selected for Local Measures. Where this is the case, the principal, not the committee, selects the baseline. Schools that select school-based/created baselines must use goal setting for that measure. CFN 204 · Diane Foley · Network Leader 14

Target Population Options Target population options include: Individual only those students that a teacher teaches (only applicable if a teacher administers an assessment to his/her own students). Grade students in a given grade level. School all students within the school. Where are they available? All 3 options – Individual, Grade, and School – are available for the state tests for testing grade teachers. For non-testing grades teachers, the only options available for state tests are grade and school. The only population available for NYC Performance Assessments is individual. For 3 rd party assessments, the only options are individual and school (this does not mean the assessment must be administered to all students in the school, it only means that the teacher will be evaluated based on everyone in the school who took the assessment). Nuances: In some cases these distinctions overlap. For example, for a 4 th grade science cluster teacher, her individual student population will also be grade and school if she teachers the entire 4 th grade. Requirements: For required State and Local Measures, the target population will always be the individual. Teachers who do not give approved individual classroom-level assessments will use either a grade- or a school-wide measure. If an assessment is repeated across measures, the target population must differ. If a school wants to use the same target population with the assessment, the Local Measures must focus on the lowest third of performers. You may only select the lowest 1/3 if the larger population was already selected as a measure. CFN 204 · Diane Foley · Network Leader 15

Measurement: Growth or Goal Setting Growth Model Goal Setting Gives teachers credit for the degree to which students exceed growth as compared to similar students. Teacher’s rating is based on the percentage of students who meet their target but does not consider the degree to which students fall below or exceed their target. Benefits & Challenges: § Does not introduce additional work in schools Benefits & Challenges: § Particularly valuable for teachers/schools with unique student populations or high mobility § Better able to account for unexpected outcomes resulting from unfamiliar, new assessments § Allows teachers and principals to individually tailor student goals § Growth model score results are not available until after assessments have been administered (i. e. , the following spring/summer) § Requires additional time/resources § Targets must be set early in the school year, possibly before much diagnostic info is gathered § Setting goals may be challenging if: § Teachers are not familiar with the comparability between assessments § Assessments are new or changing § Must use goal setting if using school-based/created baselines § § Does not actively facilitate brainstorming and discussions about individual student performance and goals Must use citywide assessments for baselines to select growth model CFN 204 · Diane Foley · Network Leader 16

Measurement: Growth Model Calculated by the state for 4 -8 ELA & math; calculated by the city for all other assessments including 3 rd grade ELA & math, science, NYSESLAT, regents, etc. Puts students in buckets according to 13 characteristics. Similar students are compared to each other to determine SGP. Some characteristics describe the student and some describe the classroom. Category Defining Characteristic of “Similar Students” • • • Prior year test score, same subject Prior year test score, different subject Retained in grade New to school in year other than entry year Average prior achievement and range around average prior score in student’s class/course (same subject) • Student with • Disability (SWD) • Student has an IEP SWD spends less than 40% of time in general education setting Percent SWD in student’s class Academic History: Controls for similar characteristics as the Progress Report, but in different ways. The Progress Report controls for classroom characteristics in their peer grouping and comparison. For growth scores, classroom characteristics are embedded in the student comparison “buckets”. The Progress Report accounts for student characteristics by adding weights to student scores, For growth scores, student characteristics are accounted for by putting students in buckets to compare them only to similar students. English Language Learner (ELL) Poverty • • • Student is an ELL NYSESLAT scores Percent of ELLs in student’s class • • Student poverty indicator (yes/no) Percent poverty in a student’s class CFN 204 · Diane Foley · Network Leader 17

Measurement: Growth Model Ms. Smith’s Class � � Prior Score Current Score SGP Student A 450 510 45 Student B 470 500 40 Student C 480 525 70 Student D 500 550 60 Student E 600 650 40 If we compare student A’s current score to other � students who had the same prior score (450), we can measure her growth relative to other students. We describe her growth as a “student growth percentile” (SGP). Student A’s SGP is the result of a statistical model and in this example is 45, meaning she performed better in the current year than 45% of similar � students. To measure teacher performance, we find the mean growth percentile (MGP) for his or her students. To find an educator’s mean growth percentile, take the average of SGPs in the classroom. In this case: Step 1: 45+40+70+60+40=255 Step 2. 255/5=51 Ms. Smith’s mean growth percentile (MGP) is 51, meaning on average her students performed better than 51% of similar students. CFN 204 · Diane Foley · Network Leader 18

Activity: Growth Model Understanding Check Turn and Talk to your colleagues and discuss the two scenarios: A teacher says to you: “I appreciate that the new evaluation system aims to measure growth, but my kids face so many challenges and just don’t grow as fast as other kids. I feel like I’m going to be disadvantaged by this new system. ” In 2 minutes or less, explain how using a growth model will address her concern. A teacher says to you: “I mean, we get these numbers called Student Growth Percentiles (SGPs), and we have no idea what they refer to. If Jose’s SGP is 65, does that mean he got 65% right on the test? ” In one minute or less, explain the misconception this teacher has about SGPs, and what this number actually refers to. CFN 204 · Diane Foley · Network Leader 19

MGPs and Statistical Confidence MGP Lower Limit 87 Upper Limit Confidence Range § NYSED will provide a 95% confidence range, meaning we can be 95% confident that an educator’s “true” MGP lies within that range. Upper and lower limits of MGPs will also be provided. § An educator’s confidence range depends on a number of factors, including the number of student scores in their MGP and the variability of student performance in the classroom. CFN 204 · Diane Foley · Network Leader 20

State Growth Model: Scoring and Results Most teachers scored Effective on the state growth score in 2011 -12. The results for NYC were slightly higher than the rest of the state: Highly Effective Developing Ineffective NYC Teachers (n=10, 544) Rest of State (n = 22, 585) 8% 76% 10% 6% 6% 77% 10% 6% Note: SED has stated that it expects the future distribution of ratings to remain similar to what it was in 2011 -12. CFN 204 · Diane Foley · Network Leader 21

Goal-Setting Process 1. Administer baseline assessment Baseline assessment administered (not required for all assessments). Report baseline assessment results. 2. DOE sends predicted student targets DOE sends predictions for how individual students will perform. 3. Teachers review DOE predicted targets Teachers may choose to adjust these targets based on additional information about their students. Predictions are based on baseline performance, student achievement history, and student demographic characteristics. Teachers submit student targets to principal. 4. Principals approve or adjust targets Principal (or designee) report finalized student targets. 5. Administer end of year assessment End-of-year assessment administered to students. 6. Teachers’ Ratings Teachers’ HEDI ratings are calculated with a conversion chart based on students’ performance on outcome assessments relative to their targets. CFN 204 · Diane Foley · Network Leader 22

Goal-Setting Considerations This is not the same as the goal-setting you may typically see in schools. Goals are scored against a state conversion chart which makes the target-setting process difficult and non-intuitive. Before considering goal-setting, make sure you understand the work and additional training this entails. CFN 204 · Diane Foley · Network Leader 23

50% Rule All teachers who have individual assessments (assessments for their students and subject) available must have State Measures that cover at least 50% of their students. Therefore, you may need to additional measures to get to 50%. Begin by accounting for courses that end in state exams that result in state growth scores (4 -8 ELA & Math) Next, add courses that end in state exams but do not lead to state growth scores. You must begin with the largest (highest enrollment) courses first. Finally, additional courses beginning with largest (highest enrollment) courses first. 50% of students covered is the minimum, where principals have choice, you may choose to cover a larger percentage. CFN 204 · Diane Foley · Network Leader 24

Does not introduce additional work Accurate results Strategic Default Option All Teachers Status Quo School wide CFN 204 · Diane Foley · Network Leader 25

DEFAULT Default only affects the Local Measure and applies for the entire school. When you default, all local measure decisions and required assessments (ELA NYC performance assessments) are eliminated and replaced with the default. The local measure default is: K-5 § SED provided growth scores on 4 -5 ELA & Math state tests § Lowest 1/3 growth scores for teachers who have school-wide measures for their State Measure K-8 Schools § SED provided growth scores on 4 -8 ELA & Math state tests § Lowest 1/3 growth scores for teachers who have school-wide measures for their State Measure K-3 & K-2 Schools § An average of all measures on the State side. Defaulting DOES NOT affect the State Measure. Principal’s are still required to make decisions highlighted in orange. All required assessments for the State Measure are still required. CFN 204 · Diane Foley · Network Leader 26

DEFAULT - Nuances Pending upcoming policy decisions If a committee does not make a decision for a particular grade and subject and it leaves it blank, the default option is automatically filled in and eliminates the required assessments. For grades without required Local Measure options, the committee may select the default options. If the committee selects the default options for a grade with required assessments, that means they are doing both the required assessments and the default. CFN 204 · Diane Foley · Network Leader 27

Activity: Committee Decision Check 1. Review the committee’s choices for assessment and target population for errors. § Circle any selections that are not allowable within the guidelines. § Use your graphic organizer to note the error, why it is an error (reference the rule), and what suggestions you have for making changes. 2. Review the committee’s choices for assessment and target population for broader implications. § Draw a box around any selections that ARE allowable, but might have broader implications that are worth discussing before selections are finalized. § Use your second graphic organizer to note the selection and what broader implications this selection might have. CFN 204 · Diane Foley · Network Leader 28

Activity: Problem Solving Scenarios – Part 1 Read and select 1 -2 scenarios to discuss with your colleagues. Use the template to guide your conversation. For each scenario: Describe the problem Indicate the problem type (technical, cultural, contextual, indecision) Brainstorm possible problem solving strategies Identify 3 next steps for the school CFN 204 · Diane Foley · Network Leader 29

Activity: State Measures Decisions Practice using the Mo. SL Options Template to make your decisions for state measures for your teachers. Use your Table of Organization: First, cross off any Pre-K teachers and teachers who do not teach 40% of the time. Make a note that these teachers will be rated S or U. Second, write the names of your teachers in the appropriate place in the left column of the Mo. SL Options Template. Exception: For teachers who do not have 50% or more of their students in a particular grade and subject, make a note to include them on the last page of the template. Check off the decisions you have made or are considering for your teachers. Discuss your thinking and reasoning with your colleagues. CFN 204 · Diane Foley · Network Leader 30

Activity: Connecting Mo. SL Elevator Speech – Take 1 More than ever, the success of the school will be dependent on the clarity of school leaders vision and communication. In light of the new Principal Evaluation System, school leaders ability to clearly and concisely voice their “theory of action” will largely impact their Measures of Leadership Practice. Over the course of the Summer Institute and throughout the year we will revisit the idea of the elevator speech as a way of working through how to best articulate the work we do. Now, as a way to synthesize our thinking around Mo. SL and next steps for our school we will begin to draft our elevator speech regarding assessment in our school. Use the elevator speech template to draft your thinking around the following prompts: Goal & Action Steps: We are working on…by…. Data & Purpose: because…so that…. . Monitoring & Next Steps: The evidence we are looking for…. based on this data our next steps will be CFN 204 · Diane Foley · Network Leader 31

Activity: Connecting Mo. SL Considering our Community Even though we may hold particular beliefs and a vision for assessment in our schools, depending on how your committee and teachers are processing this work, it may make sense to modify our decisions to honor/account for their perspective. Turn and Talk to your colleagues and share: How your teachers are reacting to or may react to this work Does that align or not with where you are in this work How might that impact your final decisions CFN 204 · Diane Foley · Network Leader 32

Activity: Connecting Mo. SL Elevator Speech - Take 2 Based on your considerations for your community, revise your elevator speech in “take 2” on the template. How might this message impact your… Meetings with your Mo. SL Committee September faculty conferences Initial planning conferences Turn, Talk & Share with your colleagues. CFN 204 · Diane Foley · Network Leader 33

Activity: Problem Solving Scenarios – Part 2 Use the Consultancy Protocol to discuss a challenge you are having at one of your schools. Protocol overview: § Presentation: Presenter provides overview of work and 1 -3 questions (3 minutes) § Clarifying Questions: Colleagues ask presenter clarifying questions (3 minutes) § Probing Questions: Colleagues ask presenter probing questions (3 minutes) § Non-presenter Discussion: Colleagues discuss the presenters work and questions without the presenter (5 minutes) § Presenter Response and Open Discussion: Presenter responds and discusses work with colleagues (5 minutes) § Reflection/Debrief: Group reflects on the process (5 minutes) CFN 204 · Diane Foley · Network Leader 34