Spotlight on Success in Everett Washington Program Evaluation

Spotlight on Success in Everett, Washington: Program Evaluation: Fidelity and Impacts Read 180 National Summer Institute San Diego, CA July 10, 2008

Spotlight on Success: Everett Public Schools Curriculum and Assessment Department Jeanne Bauer Curriculum and Assessment Specialist jbauer@everettsd. org / Telephone : (425) 385 -4053 Peter Hendrickson, Ph. D. Assessment, Research and Evaluation Specialist phendrickson@everettsd. org / Telephone: (425) 385 -4055 Tasha Lewis Instructional Facilitator, Learning Support Programs tlewis@everettsd. org / Telephone: (425) 385 -4071 2

Spotlight on Success: Everett Public Schools READ 180 Instructional Model A Community of Learners Daily Use of READ 180 Software Direct Instruction Daily Modeled and Independent Reading Practice 3

Spotlight on Success: Everett Public Schools Single Period Modification 20 minutes* Instructional Software Whole-Group Instruction 5 minutes* Small-Group Instruction 20 Modeled and minutes* Independent Reading Whole-Group Wrap-Up 5 minutes* 20 minutes* 4 *Students complete two Small-Group rotations every day along with either Whole-Group Instruction or Whole-Group Wrap-Up.

Blue Group Everett Public Schools’ READ 180 Model A=Independent Reading B=Small Group C=Instructional Software 5

Spotlight on Success: Everett Public Schools Program Evaluation Design ü Program Model ü Measure Fidelity of Implementation ü Measure Impacts Small-Group Instructional Software Modeled and Independent Reading 6

Spotlight on Success: Everett Public Schools What is Fidelity of Implementation? The extent to which the enacted program is consistent with the intended program model… --Century, Freeman & Rudnick, 2008 ü Identify Critical components ü Measure components 7

Spotlight on Success: Everett Public Schools Critical Components: Structure ü Enroll targeted students ü Daily access to R 180 software ü Text materials available when needed ü Attend Day 1, Day 2 training ü Participate in Professional Learning Community (PLC) / 7 meetings a year ü Instructional Facilitators monitor 8

Spotlight on Success: Everett Public Schools Critical Components: Instruction Teacher assures: ü Room with three areas ü Two rotations per week per student • Whole group instruction • Modeled and independent reading • Small group instruction • Read 180 Software ü Four or five r. Book Workshops per year ü Three SRI tests & Response to data 9

Spotlight on Success: Everett Public Schools Critical Components: Instruction Student completes: ü 36 (Gr 6) to 15 (HS) books read per year ü 600 to 800 minutes on software ü Attendance 95% or higher 10

Spotlight on Success: Everett Public Schools READ 180 Faculty Profile READ 180 Teacher Profile All MS HS READ 180 Teachers Number of READ 180 Teachers 33 21 12 Number of Teachers with a Reading Endorsement 6 6 0 Average Numbers of Years Teaching READ 180 1 1. 5 2 Average Number of Years Teaching 12 14 7 11

Spotlight on Success: Everett Public Schools Students Served Through READ 180 All Middle High Students Schools (9 Schools) (5 Schools) (4 Schools) General Education (Reading Support) 337 216 121 English Language Learners 105 63 42 Resource Room (Special Education) 181 120 61 Self-Contained (ERR and PBS) 98 74 24 Number of Students Served 721 473 248 12

Spotlight on Success: Everett Public Schools Regular Ed Entrance Criteria Regular Ed Exit Criteria Student has not met standard on the Reading WASL Student has met standard on the Reading WASL Student’s overall GPA is below 2. 0 Student’s overall GPA is at or above 2. 0 Student’s grade in English or Language Arts is a “D” or “F” Student’s grade in English or Language Arts is a “C” or above Student’s Lexile score indicates he/she is Student’s Lexile score consistently significantly below grade level indicates he/she is at or above proficiency grade-level proficiency English or Language Arts teacher recommendation READ 180 teacher recommendation 13

Spotlight on Success: Everett Public Schools Implementation Model Fidelity to Model Implementation Teams Program Evaluation Tools On-going Staff Development 14

Spotlight on Success: Everett Public Schools Implementation Ø Ø Central Office Teams Site-based Teams District-wide Teams Washington State READ 180 User Group Fidelity to the Model Implementation Teams 15

Spotlight on Success: Everett Public Schools Implementation Ø Ø Classroom Observations Zoomerang Surveys Principal Interviews Student Performance Data Fidelity to the Model Program Evaluation Tools 16

Spotlight on Success: Everett Public Schools Implementation READ 180 Implementation Survey 17

Spotlight on Success: Everett Public Schools Implementation READ 180 Zoomerang Surveys 18

Spotlight on Success: Everett Public Schools Implementation READ 180 Zoomerang Surveys 19

Spotlight on Success: Everett Public Schools Implementation READ 180 Zoomerang Survey Results Use of Teaching Vocabulary Red Routine Percent of Respondents 90% 80% 70% 60% Daily Several Times a Week Several Times a Month Almost Never 50% 40% 30% 20% 10% 0% Sept. 2007, n=17 Jan. 2008, n=23 May 2008, n=31 Time Period of Zoomerang Survey 20

Spotlight on Success: Everett Public Schools Implementation Frequency of Use of Red Routine: Teaching Vocabulary 90% Percent of Respondents 80% 70% 60% Daily Several Times a Week 50% Several Times a Month Almost Never 40% Never 30% 20% 10% 0% Sept. 2007, n=17 Jan. 2008, n=23 May 2008, n=31 Time Period of Zoomerang Survey 21

Spotlight on Success: Everett Public Schools Implementation Use of Think-(Write)-Pair-Share Red Routine 60% Percent of Respondents 50% 40% Daily Several Times a Week Several Times a Month Almost Never 30% 20% 10% 0% Sept. 2007, n=17 Jan. 2008, n=23 May 2008, n=30 Time Period of Zoomerang Survey 22

Spotlight on Success: Everett Public Schools Implementation Ø Ø Ø Job-embedded Staff Development Scholastic and In-house Trainings Site-based Team Meetings Teacher to Teacher Observations Monthly PLC Meetings Fidelity to the Model On-going Staff Development 23

Spotlight on Success: Everett Public Schools READ 180 Implementation Calendar 24

Spotlight on Success: Everett Public Schools Engaging the Stakeholders Administrative Leadership ü Administrative READ 180 Training ü Ongoing Principal Interviews ü Monthly READ 180 Updates in Principal Packets 25

Spotlight on Success: Everett Public Schools Engaging the Stakeholders Benefits to Administrative Leadership ü Exercise greater care in selection, support, and supervision of faculty ü Access to those students not making standard ü Stoke appetite for frequent data to monitor the program ü Provide tools to help with informal observations üPromote fidelity to the model through better understanding 26

Spotlight on Success: Everett Public Schools Engaging the Stakeholders Engagement Benefits to Managers ü Engage principal in pursuit of fidelity to the model ü Raise expectations for other programs being implemented ü Grow a culture of program evaluation ü Stay in touch with building concerns from building leadership perspective ü Gather unique ideas for improving the program ü Demonstrate caring for administrative leadership needs and concerns 27

Spotlight on Success: Everett Public Schools READ 180 Impacts ü Scholastic Reading Inventory scores ü SRI relation to WASL scores ü Expected SRI growth ü Software Usage ü Proficiency Summary ü Results Summary ü Attendance ü Disaggregated Scores 28

Spotlight on Success: Everett Public Schools Scholastic Reading Inventory (SRI) Scores and Lexiles ü Computer adaptive ü Reading comprehension ü Untimed ü Lexile Scores • Reliability…Error individual • Content Validity…real world passages • Construct Validity • Increase over time • Correlate other measures 29

Lexile Score Grade Level Comparison Grade At Risk Basic Proficient Advanced 1 N/A BR=99 - 100 -400 401+ 2 99 - 100 -299 300 -600 601+ 3 249 - 250 -499 500 -800 801+ 4 349 - 350 -599 600 -900 901+ 5 449 - 450 -699 700 -1000 1001+ 6 499 - 500 -849 800 -1050 1051+ 7 549 - 550 -849 850 -1100 1101+ 8 599 - 600 -899 900 -1150 1151+ 9 649 - 650 -999 1000 -1200 1201+ 10 699 - 750 -1024 1025 -1250 1251+ 11 799 - 800 -1049 1050 -1300 1301+ The READ 180 Student Reading Inventory (SRI) reports Lexile scores. 30

On-the Job Lexile Requirements National Adult Literacy Study Joyce and Showers (1996) Student Achievement Through Staff Development 31

Spotlight on Success: Everett Public Schools Scholastic Reading Inventory (SRI) Reporting SRI Results Criterion Referenced • 75% comprehension when score=difficulty • May set local cut points Performance Standard Reading Level Below Basic Significantly Below Grade Level Basic Below Grade Level Proficient Grade Level Advanced Above Grade Level 32

Spotlight on Success: Everett Public Schools Scholastic Reading Inventory (SRI) Norm-Referenced Results ü Percentiles (1 to 99, not equal intervals) ü NCEs (Normal Curve Equivalents) ü Stanines (Standard Nine Distribution) ü Scores BR to 1500+ ü Scores match texts…confidence and control for readers 33

Spotlight on Success: Everett Public Schools Limitations of SRI Growth Data ü Conclusions are drawn based on students with Baseline & End-of-Year SRI scores ü No control group for comparison • 85 -90% of students who qualify for the program are in the program • Students outside of the READ 180 program are not tested ü Relatively small numbers of students 34

Spotlight on Success: Everett Public Schools READ 180 Student Baseline Reading Gap and WASL Scores (Students with Baseline SRI and WASL Scores 2007 -08) 35

Spotlight on Success: Everett Public Schools READ 180 Evidence of Impact (Students with Baseline & EOY SRI/Lexile Scores 2007 -08) Scholastic Reading Inventory (SRI) Baseline and End-of-Year Data Middle Schools (5 Schools) High Schools (4 Schools) Baseline End of Year READ 180 Average Gap between Reading Level and Grade Level -4. 3 -3. 4 -6. 7 -5. 7 READ 180 Median Gap between Reading Level and Grade Level -5 -4 -7 -6 READ 180 Average Lexile Score 461. 8 574. 8 464 569. 6 READ 180 Median Lexile Score 483 632. 5 502 610. 5 36

Spotlight on Success: Everett Public Schools Program Comparisons (Students with Baseline & EOY SRI/Lexile Scores 2007 -08) Programs All Students Middle School High School n=444 n=334 n=110 Average Lexile Growth 111 113 105. 6 Average Growth in Reading Levels (Based on Lexile Scores) 0. 9 Regular Ed/Reading Support n=144 n=107 n=37 Average Lexile Growth 156. 7 164. 2 135 1. 5 Special Education n=172 n=141 n=31 Average Lexile Growth 100. 5 105. 6 76. 6 Average Growth in Reading Levels (Based on Lexile Scores) 0. 75 0. 8 0. 47 ELL n=17 n=16 n=1 Average Lexile Growth 39. 3 47. 2 -79 Average Growth in Reading Levels (Based on Lexile Scores) 0. 43 0. 47 0 Average Growth in Reading Levels (Based on Lexile Scores) 37

Spotlight on Success: Everett Public Schools READ 180 Software Usage and SRI Comparisons (Students with Baseline & EOY SRI/Lexile Scores 2007 -08) READ 180 Segments Completed Average Lexile Growth Average Growth in Reading Levels (Based on Lexile Scores) READ 180 Total Minutes 0 -10 (n = 287) 11 - 20 (n = 112) 21 - 30 (n = 30) 31 - 40 (n = 11) 41 - 50 (n = 4) 101. 2 138. 7 106. 5 38. 7 111. 2 0. 9 0. 53 0. 27 0. 9 0 - 500 (n = 192) 501 -1000 1001 - 1500 1501 - 2000 2001 - 3000+ (n = 159) (n = 61) (n = 20) (n = 5) (n = 1) Average Lexile Growth 101 125. 4 131 59. 9 -11 -150 Average Growth in Reading Levels (Based on Lexile Scores) 0. 8 1 0. 92 0. 38 0 -1 38

Spotlight on Success: Everett Public Schools Demographic Comparisons (Students with Baseline & EOY SRI/Lexile Scores 2007 -08) Gender Average Lexile Growth Average Growth in Reading Levels (Based on Lexile Scores) Meal Status Average Lexile Growth Average Growth in Reading Levels (Based on Lexile Scores) Ethnicity Males (n=262) Females (n=182) 112. 8 108. 9 0. 9 Yes No (n=291) (n=153) 95 142 0. 75 1. 2 Asian (n=53) Black (n=38) Hispanic White (n=96) (n=233) Other (n=22) Average Lexile Growth 99. 5 61 87. 9 123. 9 189. 3 Average Growth in Reading Levels (Based on Lexile Scores) 0. 71 0. 47 0. 67 1 1. 4 39

Spotlight on Success: Everett Public Schools Years of Teaching Compared to SRI Growth (Students with Baseline & EOY SRI/Lexile Scores 2007 -08) Number of Years Teaching 1 Year Or Less (n=90) 2 - 5 Years 6 -10 Years 11 -20 Years (n=59) (n=13 (n=46) 20+ Years (n=59) Average Lexile Growth 54. 7 120. 1 108 139. 9 94. 6 Average Growth in Reading Levels (Based on Lexile Scores) 0. 6 0. 9 0. 7 1. 2 0. 7 Number of Years Teaching READ 180 Average Lexile Growth Average Growth in Reading Levels (Based on Lexile Scores) 1 Year or Less (n=370) 2 Years (n=37) 3+ Years (n=37) 107. 8 134. 5 121. 8 0. 9 1 1 40

Spotlight on Success: Everett Public Schools Teacher Background Compared to SRI Growth (Students with Baseline & EOY SRI/Lexile Scores 2007 -08) Teacher with Reading Endorsement Yes (n=90) No (n=354) Average Lexile Growth 100. 3 113. 9 Average Growth in Reading Levels (Based on Lexile Scores) 0. 76 0. 92 Teacher Attended Everett Public Schools' READ 180 PLC Meetings (7 Meetings Total) Did Not Attend Meetings (n=59) Attended 1 -3 Meetings (n=37) Attended 4 -6 Meetings All 7 (n=56) Meetings (n=294) Average Lexile Growth 110. 9 61. 9 149. 9 109 Average Growth in Reading Levels (Based on Lexile Scores) 0. 82 0. 4 1. 1 1 41

Spotlight on Success: Everett Public Schools READ 180 Students Who Met Standard on the 2008 WASL (Students were enrolled in READ 180 from September 2007 to June 2008) 42

Spotlight on Success: Everett Public Schools Lessons Learned To Date ü ü ü ü ü Collect, display and reflect on data no less than every month Classroom and school collisions with our model are normal Survey and disaggregate Extract from SAM Visit classrooms, earn trust Listen to principals face-to-face Meet weekly as problem-solving group Look hard at the outliers Collaborate regionally and nationally Act on your data 43

Spotlight on Success: Everett Public Schools Thank you! Questions? 44

Everett Public Schools Acknowledgements ü 33 Middle and High School Teachers ü Ric Williams, Michelle Sekulich, Milissa Jones and “Uncle Terry” ü Everett Instructional Facilitators for Literacy ü Abba and Rider, Spot and Mr. Snuggles 45