Impact of the California High School Exit Exam

Impact of the California High School Exit Exam (CAHSEE): A Snapshot of Effects Wade Buckland, Sheila Schultz, and Becky Dvorak June 21, 2010 Presentation to The Council of Chief State School Officers National Council on Student Assessment

Background • CAHSEE Background – Exit exam assesses English language arts (ELA) and mathematics – Hum. RRO has been the independent evaluator since 1999 – A requirement to graduate since 2006 • Purpose of instruction study – Assess changes in curriculum and instruction associated with CAHSEE requirement – Identify effective intervention/remedial programs and practices that help struggling students pass CAHSEE requirements • Online surveys administered end of 2008 -09 school year – – – Principals ELA and mathematics department heads ELA and mathematics teachers Teachers of English learners (EL) Teachers of students with disabilities (SWD) 2

Survey Sample by Position Faculty Position Respondents Notes Principals 136 ELA Dept. Head 138 Math Dept. Head 162 ELA Teacher 159 (1 -6 surveys per school) Math Teacher 178 (1 -5 surveys per school) EL Teacher 95 (1 -3 surveys per school) Teacher of SWD 126 (1 -4 surveys per school) Total 994 • Targeted 400 schools • At least one faculty member responded from 271 different schools 3

Presentation Focus • Teacher Quality: How do teacher experience and teaching credentials relate to student performance on the CAHSEE? • Standards Integration: How have California’s content standards been integrated? • Student Remediation: Have students had access to remediation? If so, what impact has the remediation had on student performance? • Impact on Instruction: How has the CAHSEE requirement influenced the California school system? 4

Teacher Quality: Years Teaching Teachers with Five or More Years of Experience Primary or Supplemental Courses Intervention Programs ELA (n=105) Math (n=135) ELA (n=104) Math (n=135) Most 67 67 59 59 Almost Half 19 24 14 14 Few 11 8 13 13 Not Applicable 4 2 15 14 • Teachers were relatively experienced. - The majority of Department heads reported that most of their teachers had at least 5 years of experience for ELA and Math -More common that most teachers teaching primary supplemental courses had at least 5 years experience than teachers of intervention programs 5

Teacher Quality: Student CAHSEE Performance Relationship of Teaching Experience to CAHSEE Performance of 2009 10 th Grade Students Years of Teaching Experience: Primary and Supplemental Courses Statistical Finding Significant Relationship Years of Teaching Experience: Intervention Programs ELA Math f(2, 96)= 4. 10, p<. 05 f(2, 127)= 1. 71, p>. 05 f(2, 84)= 4. 85, p<. 05 f(2, 111)= 2. 39, p>. 05 Yes No • ELA: Years of teaching experience was positively related to student CAHSEE performance for primary and intervention courses. • Math: No significant relationships. • Further analyses found no significant relationship for ELA or math between teaching experience and gain in CAHSEE score from Grade 10 to Grade 11 (p>. 05), regardless of course type. 6

Teacher Quality: Credentials Full Teaching Credential Prevalence Department Head Response ELA and Math Teachers Who Worked With a Full Credential Intervention Programs Taught by Teachers With a Full Credential ELA (n=105) Math (n=136) ELA (n=104) Math (n=132) Nearly or All (more than 90%) 74 45 68 54 Most (75 -90%) 14 29 11 10 Some (25 -74%) 7 22 10 17 Only a few (less than 25%) 3 3 5 11 None 2 2 7 8 • ELA classes tend to have more teachers working with an appropriate credential than math classes. • Intervention programs are slightly more likely to be staffed by teachers not holding an appropriate credential than a typical ELA or math class. 7

Teacher Quality: Student CAHSEE Performance Relationship of Teaching Credential to CAHSEE Performance of 2009 10 th Grade Students How many teachers in your department have a full ELA/math credential? Statistical Finding Significant Relationship How many of your intervention programs/courses are taught by fully credentialed ELA/math teachers? ELA Math f(5, 97)=5. 69, p<. 05 f(5, 127)=5. 20, p>. 05 f(5, 96)=2. 99, p<. 05 f(5, 123)=1. 20, p>. 05 Yes Yes No • ELA: Having appropriate teaching credential was positively related to student CAHSEE performance for primary and intervention courses. • Math: Having appropriate credential was positively related to performance for primary courses; no significant relationship for intervention courses. • Further analysis found no significant relationship between teaching credential and gain in CAHSEE score from Grade 10 to Grade 11. 8

Standards Integration: CAHSEE Blueprints Principals Reporting Completion of School’s Integration of California Content Standards Contained in Blueprints Adopted for CAHSEE by Year Response Percentage ELA 2004/05 (n=227) • • Math 2008/09 (n=135) 2004/05 (n=227) 2008/09 (n=135) Complete (96 -100%) 47 57 49 53 Mostly Complete (61 -95%) 47 39 43 41 Partially Complete (40 -60%) 6 2 7 4 Little (less than 40%) 0 1 1 1 Do not know 0 2 In 2005 and 2009, reported integration was mostly complete or complete. In 2009, a slightly higher percent reported integration was complete compared to 2005. 9

Student Remediation: Courses Offered CAHSEE Intervention/Remediation Courses Offered, According to Principals Response Percentage (n=130) Yes, for sophomores, juniors, and seniors 39 Yes, for juniors and seniors 48 Yes, for seniors only 2 We do not have CAHSEE intervention courses 11 • • Almost half of responding principals indicated their schools offer remediation courses targeted to juniors and seniors who have struggled to pass the CAHSEE requirements. Approximately 11% of responding principals indicated their schools do not offer remediation courses. 10

Student Remediation: Impact on CAHSEE Scores Effectiveness of District in Improving CAHSEE Scores for At-risk Students, According to Principals Response Percentage (n=120) Minimally Effective 26 Effective 42 Very Effective 21 Extremely effective 12 • About two-thirds of responding principals indicated their schools’ remediation efforts have been at least minimally effective in improving students’ CAHSEE scores. 11

Student Remediation: Impact on Drop-out Impact of CAHSEE Intervention/Remediation Courses on Preventing Students from Dropping out of School, According to Principals Response Percentage (n=115) Not at all 6 Slight extent 24 Moderate extent 37 Great Extent 29 Very Great Extent 4 • Of responding principals whose schools offer remediation, almost 90% said those courses have had a positive impact on preventing student drop-out. 12

Impact on Instruction: Student Preparation Extent Increased Focus of Using CAHSEE-like Tasks, Matching Content of Instruction and Content of CAHSEE, and Covering Test-Taking Skills to Prepare Students for CAHSEE Response Using CAHSEE-like Tasks Matching Content of Instruction and Content of CAHSEE Covering Test. Taking Skills ELA % (n=153) Math % (n=169) ELA % (n=153) Math% (n=169) Not at all 5 11 1 4 Slight Extent 27 26 21 25 21 22 Moderate Extent 38 37 43 30 48 36 Great Extent 29 26 31 34 29 38 • While increased focus on using CAHSEE-like tasks, matching instruction to CAHSEE content, and covering test-taking skills to prepare students was fairly consistent, ELA teachers tended to increase their focus slightly more than math teachers, especially in matching instruction to CAHSEE content. 13

Impact on Instruction: Instigate Change Teacher’s Use of Assessment Results Response Teacher Type ELA % (n=159) Math % (n=178) Provide feedback to students 87 89 Modify lesson plans 87 85 Practice differential instruction 70 69 Assign to supplemental/remedial classes 30 31 Provide feedback to district 40 29 • CAHSEE assessment results were used at a high frequency to provide feedback to students, modify lesson plans, and practice differential instruction. • Less than half of respondents indicated that CAHSEE assessment results were used to assign students to supplemental/remedial classes and provide feedback to districts. 14

Impact on Instruction: Instigate Change Ways in Which ELA and Math Teachers Collaborated With Other Teachers Teacher Type Response ELA % (n=159) Math % (n=178) By aligning instruction across courses 57 62 By planning coverage of CAHSEE standards 50 48 By assessing individual student needs 62 51 By sharing ideas about teaching strategies 88 83 I collaborate with other teachers in another way 13 11 I do not collaborate with other teachers 3 7 • Most teachers reported they collaborate with other teachers in some way, especially by sharing ideas. • At least half of responding teachers reported collaborating by aligning instruction across courses, assessing individual student needs, and planning coverage of CAHSEE standards. 15

Impact on Instruction: Instigate Change Extent to Which CAHSEE Is Used to Change School’s Instruction, Assessment, and Overall Improvement Response School’s Instruction School’s Assessment Overall School Improvement ELA % (n=153) Math % (n=169) ELA % (n=152) Math % (n=169) ELA % (n=153) Math % (n=170) Not at all 13 22 16 21 12 12 Slight Extent 35 34 34 37 27 35 Moderate Extent 40 32 43 28 43 41 Great Extent 12 12 7 14 18 13 • Most teachers reported CAHSEE is used to a slight or moderate extent to change their school’s instruction, assessment, and overall improvement. • CAHSEE was used slightly more to make overall improvements to the school than to change instruction or assessments. 16

Summary • Teaching experience was sometimes related to student CAHSEE performance – Years of experience for ELA teachers significantly related to ELA student CAHSEE performance • CAHSEE content standards have commonly been integrated into course instruction • Remediation/Intervention programs commonly present in schools – Although most believe these programs at least somewhat improve performance and decrease drop out rates, there is room for improvement • CAHSEE found to impact classroom instruction – Teachers reported increasing their focus on using CAHSEE-like tasks and matching the content of instruction with that of the CAHSEE 17

Further Reference • Full Reports: http: //www. cde. ca. gov/ta/tg/hs/evaluations. asp • Specific questions: wbuckland@humrro. org 18