Team Read Improving Literacy in the Seattle School

Team Read Improving Literacy in the Seattle School District PA 590 C – Program Evaluation Fall 2007 Anne Debuyserie

“. . . a Seattle School district tutoring program dedicated to increasing the reading skills of elementary students through year-long coaching by high school students . . . The program's goals for participating high-school tutors are to develop work experience, a sense of responsibility and accomplishment, and to learn the rewards of community service. ” (Team Read Mission Statement, 1998)

What is Team Read? Private-public partnership to improve literacy in Seattle Public Schools n Cross-age tutoring program using high school student as tutors n Started in March 1998 with a threephase implementation n June 1999: 10 schools, 335 students, 300 coaches n June 2000: addition of 7 schools n

Team Read- Org Chart

Team Read Stakeholders Craig Mc. Caw (and wife Susan): Businessman n John Stanford: Seattle Schools Superintendent n Joan Dore: Seattle Schools Reading Specialist n Tricia Mc. Kay: Team Read Program Mgr n Team Read Advising Board: Seattle School District Staff, Alliance for Education pgm representatives, Mc Caws, community representatives. n Additional investors joined by year 2. n

Who is Team Read? n Coaches – Strict selection guidelines – High expectations from Team Read leaders – Compensation n Readers – School selection comes first – Student selection based on reading level – Eligibility based on need – First-come, first-serve basis – Low socio-economic background

Who is Team Read? n Site Coordinators – Part-time, teachers or school staff – Meet informally with Tricia Mc. Kay for feedback n Volunteers – Needed due to amount of work – Partnership with VISTA/Americorps and U. of Washington Pipeline – Assist site coordinators

Team Read Session n n Two 1 hr session per week during school year Standardized session format and structure – – 3. 00 PM: coaches arrive 3. 15 PM: snack with reader 3. 30 pm: reading session begins 4. 15 PM: reading session ends/ coaches fill out paperwork – 4. 30 PM: reading coaches leave n Opportunity for site coordinators to be creative

Team Read Evaluation Results 1 st year evaluation: not as positive as expected n 2 nd year: promising but Team Read’s impact not as promising as hoped n Tricia Mc. Kay disappointed by results and wonders if findings provide her with the information needed for improvements n Questions about the accuracy of results. n

Team Read Evaluations n n n Evaluation conducted by Margo Jones, statistician by trade. 17 schools in the program by June 2000 but only 10 used in the evaluation. 3 main objectives to show Team Read’s Impact: – Goal#1: Do the reading skills of the student readers improve significantly during their participation in the Team Read program? – Goal#2: How does the program affect the reading coaches? – Goal#3 What is working well, and what can be improved?

Evaluation Methods Goal #1 n Goal #1: Reading skills. Two-fold approach Grade Pre-Test Post-Test Metric 2 DRA Fall 1999 DRA Spring 2000 Test Level 3 DRA Spring 1999* ITBS Spring 2000* NCE 4 ITBS Spring 1999 WASL Spring 2000 Scale Score 5 WASL Spring 1999 ITBS Spring 2000 NCE

Apples and oranges Grade Pre-Test Post-Test Metric 2 3 4 5 NCE

Evaluation Methods Goal #1, Cont’d 2 nd method: determine proportion of Team Read participants who moved from below grade level test score to ator above level compared to all students from entire district n Note: one of the selection criteria to be in Team Read is to be within bottom 25% of district reading test score n

Evaluation Methods Goal #1 n n n Both methods for goal 1 are flawed. Note from Jones in report confirming problems with different tests. “The question then becomes ‘Is it reasonable to assume that the pre- and post-tests measure the same skills? ’ The evaluator adopted a correlation criterion for answering this question. If the correlation between pre- and post-tests was near or above. 8 (see footnote 5) (as was the case for the 2 nd grade, where pre- and post-tests were essentially the same), the pre- to post-test change score was interpreted as a gain score. If the correlation coefficient was much less than. 8, no analysis was performed. ”

Evaluation Methods, Goals #1 Assessment n Visible problems with tests: – Pre and post-tests do not measure the same skills except in 2 nd grade – All Team Read participants are within the bottom quartile of the reading test scores and can’t be compared with students district-wide. – Analysis could not be carried out for some grades (3 rd) due to incompatibility of reading tests

Evaluation Method, Goal #2 n n n Method: questionnaire given to coaches during their last week of coaching. Goal: to assess the impact of Team Read on coaches. Identify 3 main areas of job satisfaction: – How positive is their experience with TR? – Extent to which the coaches feel that TR helps students readers – How supportive is TR’s site coordinators and staff?

Evaluation Method, Goal #2 Assessment Jones results showed that coaches were satisfied with their job and felt a sense of pride and accomplishment. n Test performed by Jones was successful in meeting its goals, unlike in Goal #1. n Some questions provided answers as to what to improve in the program. n Test is useful to the program manager n

Evaluation Method, Goal #3 Method: interviews with major stakeholders, one site visit, review of last 5 years’ research literature on cross -age tutoring and last 3 years research on best practices. n Goal: assess the overall progress of the program. n

After the Evaluation Margo Jones makes her recommendations based on her findings to improve the efficacy of Team Read. n Tricia Mc. Kay reads the evaluation but fail to see what the results are telling her about the program n

Evaluation Methods - Summary Goals Methodology Reading improvement among elementary students n Analyzed standardized test scores for 10 Coach development n Surveyed coaches by questionnaire Program implementation n. Interviewed stakeholders of 17 Team Read schools. n Rate of increase on pre- and post- tests. n Percent of students moving from below reading level to at/above grade level. n Collected data by grade level and by school. n Compared Team Read results to district scores. n. Observed Team Read session n. Reviewed other studies

Evaluation Problems Academic v. Practitioner approach. n Result presentation. n Research design flaws. n – Not all the data was analyzed – Comparison with dissimilar items – Statistical analysis stretched Evaluation only covers 3 goals. n Lack of communication between the evaluator and the program manager. n

What Could Have Been Done Better?

Impact: 2005 -2006 Reading Results From Team Read’s 2006 Annual Report. http: //www. teamread. com/downlo ads/team_read_106903_report_pro of. pdf

Team Read Update - 2006 n Program Evaluation Results for 2005 -2006 School Year – 70% of 2 nd graders and 52% of 3 rd graders were reading at/above or approaching grade level – 54% of 2 nd graders and 35% of 3 rd graders gained greater than 1. 5 grade levels in reading – 98% of the parents of 2 nd & 3 rd graders and 92% of their teachers reported increased reading skills as a result of participation in Team Read – 76% of 2 nd & 3 rd graders said that reading was more fun since joining Team Read