Student Survey Proctor Orientation TNTP Academy and the

Student Survey Proctor Orientation TNTP Academy and the Assessment of Classroom Effectiveness January 2016 © TNTP 2013

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing /2

Through ACE, we strive to create the fullest possible picture of teacher performance using multiple measures. • Research shows classroom observations are valid predictors of teacher effectiveness. Classroom Observations • You will be observed at least 4 times, with your performance rated according to the ACE Instructional Framework. Principal Ratings • Principals will rate your performance relative to other new teachers they have worked with throughout their careers. • The survey uses research-tested questions predictive of teacher effectiveness. Student • If you work with very young students (grades K-2) or with students in Surveys moderate-to-severe special education settings, you will most likely not administer student surveys. • The best measure of your effectiveness is the academic growth of your students. Student Achievement • Where available, we will incorporate data from state-produced TVAAS Data score reports to analyze your actual impact on student learning over the course of the school year. /3

Measure 3: Student Surveys WHY? HOW? • Research from the Gates Foundation’s Measures of Effective Teaching (MET) study demonstrates that student surveys are a reliable indicator of teacher effectiveness. • Students in grades 3 -12 will take paper-based surveys. For all teachers, a proctor will administer surveys. • Our student survey uses questions shown to be predictive of teacher impact on student learning. “The average student knows effective teaching when he or she sees it. ” (“Learning about Teaching: Initial Findings from the Measures of Effective Teaching“ Gates Foundation, December 2010) • Teachers working with very young students (K-2) or in moderatesevere settings will not administer the survey. • Surveys and consent forms will be available in Spanish and English, with potential translation into other languages. WHEN? • Teachers must collect parent consent forms before students complete surveys. • Student surveys will be administered between January 25 and March 18. • Sites will be administering on a staggered schedule – Amy will communicate with proctors directly regarding the survey window for your site. • All surveys will be anonymous and must be kept completely confidential. /4

How the student survey process works 1. ADMINISTER SURVEY Students complete an anonymous, paper-based survey. The survey administration takes approximately one class period and will be administered by proctors to ensure confidentiality. 2. GENERATE SCORE 3. IMPROVE PERFORMANCE Survey questions are grouped into three broad categories: Care, Control, and Challenge, with responses rated on a 5 point scale. Teachers receive formative score reports detailing their students’ perceptions. This data provides valuable insights into strengths and weaknesses as seen by the students in our teachers’ classrooms every day. Student responses will be averaged to generate a composite score in each of these categories, and those three category scores will then be averaged to generate the overall survey score. Student Survey 1– 5 Survey Score /5

Calculating the survey score. Harry Potter Hermione Granger Ronald Weasley Cho Chang Neville Longbottom Lavender Brown Padma Patil AVERAGE CATEGORY SCORES STUDENT COMPOSITE SCORES BY CATEGORY Care Control Challenge 4 4 2. 75 3. 75 4 2. 5 4. 25 3. 5 3 4 3. 75 2. 75 4. 25 3. 25 2. 75 4. 07 3. 79 2. 82 TEACHER’S OVERALL SURVEY SCORE: 3. 56 Teacher’s overall survey scores map to the performance levels below. Skillful Proficient Developing Minimally Effective Ineffective 4. 50 – 5. 00 4. 00 – 4. 49 3. 50 – 3. 99 3. 00 – 3. 49 2. 99 or lower /6

Past participants valued student survey feedback, indicating that it would help them improve their teaching. CHALLE NGE CARE CONTRO L • “I was glad to see my students recognized the value of the homework I assigned, that they believe that I want them to explain their thinking, and that they knew that I believed that they could get a good grade if they tried. “ • “I deeply cared for my students, but I know that I didn't show them or tell them enough. This is definitely something I will focus on next year. “ • “[After reviewing survey results, ] I am going to give the kids more input into the behavior system in my classroom, in an effort to give them ownership of their own actions as individuals and as a class. ” /7

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing /8

Your performance on each ACE measure will be mapped to the performance scale below, earning you between 1 and 5 points for each ACE component. Classroom Observations Principal Ratings Student Surveys Student Achievement 1. 00 – 1. 99 points Much worse than 1 point 2. 99 or lower 1 point Ineffective 1 point Minimally Effective 2. 00 – 2. 79 points Worse than 2 points 3. 00 – 3. 49 2 points Minimally Effective 2 points Developing 2. 80 – 3. 59 points Same as 3 points 3. 50 – 3. 99 3 points Developing 3 points Proficient 3. 60 – 4. 29 points Better than 4 points 4. 00 – 4. 49 4 points Proficient 4 points Skillful 4. 30 – 5. 00 points Much better than 5 points 4. 50 – 5. 00 5 points Skillful 5 points Ineffective Each component will be weighted and combined to calculate your final ACE rating. /9

Adding it up: Four Scoring Scenarios Scenario 1: Classroom Observations + Principal Ratings Principal Rating Observations (40%) (60%) Scenario 2: Classroom Observations + Principal Ratings + Student Surveys Student Survey Observations s (40%) (30%) Principal Rating (30%) 0. 40 * 3 points = 1. 20 points (Classroom Observations) 0. 60 * 3 points = 1. 80 points + 0. 30 * 3 points = 0. 90 points (Principal Rating) (Observations) + 0. 40 * 4 points = 1. 60 points (Principal + 0. 30 * 2 points = 0. 60 points (Student Rating) Surveys) 3. 4 points ( Final ACE Score) 2. 7 points ( Final ACE Score) / 10

Adding it up: Four Scoring Scenarios, cont’d Scenario 3: Classroom Observations + Principal Ratings + Student Surveys + Student Achievement Data Student Observations Achievemen (25%) t (35%) Principal Rating Student (20%) Surveys (20%) 0. 25 * 3. 15 points = 0. 79 points (Observations) + 0. 20 * 3 points = 0. 60 points (Principal Rating) + 0. 20 * 2 points = 0. 40 points (Student Surveys) + 0. 35 * 4 points = 1. 45 points Scenario 4: Classroom Observations + Principal Ratings + Student Achievement Data Principal Observations Rating (35%) (30%) Student Achievemen t (35%) 0. 35 * 3. 15 points = 1. 10 points (Observations) + 0. 35 * 4 points = 1. 40 points (Student Achievement) + 0. 30 * 3 points = 0. 90 points (Principal Rating) / 11

There are 3 possible outcomes at the end of the first year of teaching. PASS 2. 75 – 5. 00 points • If a teacher passes ACE and complete all program and state requirements, her or she will be recommended for their state certification. EXTENSION 2. 50 – 2. 74 points • If a teacher falls short of passing, but is on the right trajectory to meet that standard with additional progress, he or she may be granted an extension year to continue building skills. REMOVAL 1. 00 – 2. 49 points • If a teacher struggles and shows limited prospect of improvement, her or she will not advance to another year of teaching. • While we anticipate this outcome to be relatively rare, it is necessary. Allowing a chronically struggling teacher to remain in the classroom isn’t good for anyone. / 12

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing / 13

Not all teachers will be eligible for student surveys. Pre-K through 2 nd grade • A valid survey instrument has not yet been developed for this age group. Moderate to severe SPED • A valid survey instrument has not yet been developed for this population. Teachers with fewer than 15 students • 15 is the minimum number of students our analysts required to produce reliable scores on student surveys for ACE. Remember that we must get 15 students in total – not just in one class period/course section! *If you find that you’ve been assigned a teacher or course sections fitting any of these categories, please let us know immediately! / 14

Teacher will administer surveys in up to two course sections. Element ary (3 -5) Seconda ry (6 -12) Exceptio ns • If teachers work in a self-contained setting, they will administer just to that class (one “course section”). If they teach in a departmentalized setting, they’ll administer to two course sections. • If teachers work in a secondary setting (typically teaching 1 -2 subjects), they’ll administer surveys to two course sections/class periods of their choosing. • Some teachers completed the Class Period Selection Tool incorrectly and only selected one – so we only printed surveys for one class period. *The number of class periods for which each of your teachers will administer surveys will be clearly marked on your caseload! / 15

What does the survey actually look like? • The surveys consists of 20 -25 questions (more for secondary). • Surveys are individually coded for each teacher. • You’ll have plenty of extras! We won’t provide blanks because of the unique ID’s. • Shred the leftovers – but not until you’ve sent everything back to the Brooklyn office! / 16

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing / 17

Student Survey Windows Site Window Opens (Start Surveying) Window Closes By when should I ship all surveys back? Arizona February 24 March 18 (preferably earlier) Baltimore February 24 March 18 (preferably earlier) Charlotte February 24 March 18 (preferably earlier) DC February 24 March 18 (preferably earlier) Fort Worth February 24 March 25 Return to site for shipping Indianapolis February 24 March 18 (preferably earlier) Louisiana January 25 March 18 (preferably earlier) Nashville February 3 March 4 (preferably earlier) / 18

Overview of what teachers have done so far… Ø Teachers have already given us their course selections via the Class Period Selection Tool in Survey Monkey. These data were used to print and ship surveys for each teacher to your site. Ø Teachers have also included information about the number of foreign language and large print surveys they need, as well as whether students in their selected classes will require a readaloud accommodation. Ø Teachers are now in the process of collecting permission forms from students in order to have them take the survey – this is hard work, and you’ll want to be careful not to visit a teacher if they haven’t collected enough forms yet! Ø Some teachers on your caseload have not given us any information about their course selections yet, so we’ve guesstimated how many surveys they’ll need. You should reach out to them directly for specific course meeting times. / 19

Scheduling Survey Sessions Receiving your caseload from the site o You’ll access your caseload via a dynamic spreadsheet in Google Drive after receiving an introductory email from us. o If you’re Arizona, Baltimore, Charlotte, DC, Indianapolis, Louisiana, Michigan, or Nashville, you’ll be reporting directly to me/Janet. I’ll be sharing your caseload and asking you to report in regularly via a Google doc. Accessing teacher course selections and planning your schedule o Once you’ve received your caseload via the Google doc, you’ll see that it catalogs everything you need to know to schedule with your teachers - including the time each selected class period meets. Tips and tricks for scheduling include: § Allowing time for rescheduling – try to front-load your window § Emailing teachers without class period meeting times early and following up strongly § Using www. batchgeo. com to see your geographic layout § Sending a confirmation email 1 -2 days before your visit! Confirming appointments with teachers o Once you’ve planned your schedule, add these dates to the Google doc. o Email teachers directly using the email templates provided on the wiki! Tracking and reporting your schedule o Please think of the Google doc as a living record – keep this up-to-date with daily revisions to confirmed meeting times, reschedules, # of surveys collected, etc. . / 20

Receiving Surveys You’ll receive your packet of surveys via Fed Ex to your home address. Louisiana, Nashville, Indy, and Forth Worth teachers will receive surveys no later than the end of next week. Arizona, Baltimore, Charlotte, Michigan, and DC proctors will receive them closer to the start of their survey windows. You’ll return your completed surveys to our Brooklyn office using the pre-paid mailer, which will also arrive at your doorstep before the end of the survey window. 1 -2 Weeks Before Surveys Start / 21

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing / 22

Survey administration will happen in three segments. Scheduling Administration Report and Return / 23

Survey Administration Workflow 1 2 3 4 • Arrive at the school at least 15 minutes early. Carry your TNTP identification (if provided). • Check in at the front desk. • Proceed directly to the classroom. • Introduce yourself to the teacher and walk them through the survey process. 5 • Introduce yourself to the class and distribute the surveys. Ensure that students keep the survey upside down until you give them permission to start. 6 • Read the survey script word for word before allowing students to turn over the survey and begin answering the questions. 7 8 9 • Have students turn the survey over when they’re done. • Collect the surveys. Thank the teacher on your way out. • Update the Google doc with the total number of surveys collected that evening. / 24

Consent Forms Teacher collects permission slips Teacher notifies Proctor of which students are eligible Proctor administers surveys ONLY to approved students / 25

Survey Script Please pay close attention while I share important information about the survey that you are about to take. TNTP, a national nonprofit organization with more than 15 years of experience conducting education surveys across the country, wants to hear from you about your experiences in this class. The reason you are taking this survey is to learn what is working well and what needs improvement within your classroom. There are no right or wrong answers. This is your chance to provide your honest feedback. Your answers will be used to better understand your teacher’s strengths and weaknesses from students’ point-of-view. You may choose to skip any questions, but I hope that you will take this chance to share as much of your thoughts as possible. Your responses are completely confidential. Your responses will be combined with responses from other students in your class before being shared with your teacher. Your teacher will not see how you or any other individual student in this class responded, so please feel safe to answer these questions honestly. When you are completing the survey, please fill in the circles completely that represent your answer. I cannot answer any questions about individual items on the survey, but please pick the answer that you think best represents the way you feel about your experience in this classroom with your teacher. This is not a long survey, so please take your time in completing it so that you can share your honest and thoughtful responses. Once you have your copy of the survey, you may begin. Please note that you are not being asked to share your name, so please do not write in your name on the form. Once you have finished, please hold onto your survey form until everyone in the class is finished. I will collect them after all students have completed the survey. / 26

Survey Script – Additional Language for Co-Teachers When two ACE participants from the same classroom are surveyed: First Administration: • This class has two teachers, so you will take the survey two times – one time for each teacher. When you take the survey today, only think about insert teacher’s name. Tomorrow you will be able to fill out a survey just for about insert other teacher’s name. Second Administration: • This class has two teachers, which is why you are filling out this survey again. When you took the survey yesterday, you thought about insert teacher’s name. When you take the survey today, only think about insert other teacher’s name. When only one co-teacher from the same classroom is surveyed: • Even though this class has two teachers, this survey is only for insert teacher’s name. When you take the survey today, only think about insert teacher’s name. *These scripts will be posted to the wiki for easy reference, along with all email templates, key dates, and other resources! / 27

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing / 28

Accommodations You may be required to provide accommodations for one or more students during survey administration. Accommodations can take the following forms: Large Print Read Aloud/Scribe Language Other Than English Any required accommodations will be noted in your schedule. If you are required to read aloud to a student, you should take him/her to a quiet corner of the room so as not to disturb other students. If the majority of students require read-aloud accommodation, you may consider reading to the full class. / 29

Professional Responsibility for Proctors Surveys are a key component of our teachers’ summative evaluation (ACE) and directly impact their ability to be certified. You are responsible for maintaining the integrity of these surveys, both before and after administration, until they are returned to our Brooklyn office. High-Stakes Student Surveys Please return all surveys at once you will receive multiple mailers, but send them all at the same time. Be sure to include a cover sheet for each mailer. Please schedule return pick-up of your surveys for the day after your last survey session. Timeliness is essential! / 30

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing / 31

Some Key DO NOTS: DO NOT administer surveys to students without permission forms. DO NOT mix survey types for one teacher (i. e. , a teacher cannot give the elementary survey to a 5 th grade section and a secondary survey to a 6 th grade section – he/she must choose all secondary or all elementary sections). DO NOT use blank copies of the survey or copy one of your existing surveys if a class section is short materials- reach out to Amy for extra surveys that are individually coded. DO NOT allow the teacher to circulate or answer questions during survey administration! / 32

What If…. The teacher isn’t teaching that day? • If the substitute or co-teacher is willing to have you administer the survey, then proceed, If not, then reschedule. A student asks me for help on a question? • You cannot help a student interpret a question – say “there’s no right or wrong answer, so please just answer the question as best you can. ” The class has fewer than 15 students? • If this is a self-contained elementary classroom, you’ll need to reschedule until the teacher has collected enough permission forms or enough students are present. If this is a secondary classroom, ensure that you’ll be able to survey 15 students total between the two classes. Some students don’t have permission slips on file? • We cannot survey students without a permission slip. These students should not leave the room, but they should not be given a survey. The teacher is responsible for indicating which students have permission and which do not. / 33

What If…. (cont. ) I don’t have enough surveys for all students with a permission slip? • This should not happen – we print surveys for the total number on the roster, not the number of permission forms teachers report collecting! I’m running late and class has already started? • Please try to let the school know from the road if you’ll be late. Only administer surveys if you’re confident students will have enough time to complete the survey thoughtfully and without rushing. The teacher tries to turn me away? • Teachers have confirmed dates that fit their schedule, so this should happen very rarely. If the teacher attempts to turn you away, insist on continuing or call Amy/Janet for back-up. Students opt not to answer all of the questions? • Students can answer as much or as little of the survey as they’d like, but encourage them to complete the survey fully. / 34

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing / 35

Billable Hours Non-Billable • Hours in the classroom proctoring surveys • Time spent in the Google doc, planning your schedule, and communicating with teachers • Drive time IN BETWEEN schools • Time spent planning for rescheduled survey sessions • Drive time to your first school or home from the last school • Lunch breaks • Extended breaks between survey sessions due to inefficient scheduling or rescheduling / 36

Replicon / 37

Contents Why We Care About Effective Teaching Understanding ACE Adding It Up: ACE Scoring Student Survey Parameters Scheduling Survey Sessions Survey Administration Accommodations Troubleshooting Compensation Closing / 38

Questions? / 39