Evaluating Outcomes of the English for Speakers of

  • Slides: 16
Download presentation
Evaluating Outcomes of the English for Speakers of Other Languages (ESOL) Program in Secondary

Evaluating Outcomes of the English for Speakers of Other Languages (ESOL) Program in Secondary Schools: Methodological Advance and Strategy to Incorporate a Quasiexperimental Design Helen Wang, Ph. D. & Shahpar Modarresi, Ph. D. Montgomery County Public Schools Rockville, Maryland [email protected] org http: //www. montgomeryschoolsmd. org/departments/ sharedaccountability/

Program Overview n n Purpose: Help ESOL students function linguistically and culturally in regular

Program Overview n n Purpose: Help ESOL students function linguistically and culturally in regular classrooms Participants: Students speaking a language other than : standard American English Length: Throughout school year Curricula: Rigorous standard-based ESOL curriculum : Outcomes: Improvement and Attainment of English mprovement language proficiency and achievement in content areas

Evaluation Questions 1. To what extent did secondary ESOL students make annual progress toward

Evaluation Questions 1. To what extent did secondary ESOL students make annual progress toward the state target for acquiring English language proficiency (the percentage of students gaining a 15 -point or more increase annually in overall scores of Language Assessment System Links or LAS-Links)? 2. To what extent did secondary ESOL students attain English language proficiency toward the state target (the percentage of students achieving the advanced level in overall LAS-Links and the proficient or advanced level in subtests)? 3. How did Grade 8 ESOL students perform on Measures of Academic Progress– Reading (MAP-R)? 4. How did Grade 8 ESOL students perform on Maryland School Assessments (MSAs) in reading and mathematics? 5. Were students receiving higher-level ESOL instruction more likely to pass High School Assessments (HSAs) than those receiving lower-level ESOL instruction?

Evaluation Design n Incorporated Quasi-experimental design Specified samples across years to examine trends of

Evaluation Design n Incorporated Quasi-experimental design Specified samples across years to examine trends of student performance Specified outcome measures and analytical procedures pertaining to research questions

Outcome Measures: Standardized Assessments n n Language Assessment System Links (LAS-Links) overall and subtest

Outcome Measures: Standardized Assessments n n Language Assessment System Links (LAS-Links) overall and subtest scale scores and proficient levels Measures of Academic Progress Assessment in Reading Rasch Unit (MAP-R RIT) scores Maryland School Assessment (MSA) reading and mathematics scale scores High School Assessment (HSA) passing status in algebra, English, biology, and government

Study Samples for Evaluation Questions n n n Q 1: Grade 6 -12 ESOL

Study Samples for Evaluation Questions n n n Q 1: Grade 6 -12 ESOL students with LAS-Links scores in any two adjacent years of most recent three school years. Q 2: Grade 6 -12 ESOL students with LAS-Links scores in the most recent two school years Q 3: Grade 8 students with MAP-R RIT scores in most recent three school years Q 4: Grade 8 students with MSA reading or math scale scores in most recent three school years Q 5: English learners from two most recent graduating classes who received ESOL instruction at the time they maximized an HSA score.

Analyses for Question 1: To what extent did secondary ESOL students make annual progress

Analyses for Question 1: To what extent did secondary ESOL students make annual progress toward the state target for acquiring English language proficiency? Performed descriptive procedures on LAS-Links for the three selected years: n n Computed numbers and percentages of students who gained 15 or more points on the overall score from the previous year, by grade level, ESOL instructional level, demographics, and service receipt measure. Compared the percentages with state targets which increased over the years

Analyses for Question 2: To what extent did secondary ESOL students attain English language

Analyses for Question 2: To what extent did secondary ESOL students attain English language proficiency toward the state target? Performed descriptive procedures on LAS-Links for the two selected years: n n Computed numbers and percentages of students who achieved the advanced level overall and proficient or advanced level for speaking, listening, reading, and writing subtests, by grade level, ESOL instructional level, demographics, and service rec Compared the percentages with state targets which increased over the years

Analyses for Questions 3 and 4: 3. How did Grade 8 ESOL students perform

Analyses for Questions 3 and 4: 3. How did Grade 8 ESOL students perform on MAP-R? 4. How did Grade 8 ESOL students perform on MSA reading and math? Performed analyses of covariance (ANCOVA) for cross-year comparisons on MAP-R RIT scores and MSA reading and mathematics scale scores for the three selected years; computed and used propensity scores as a covariate three selected years; c 2007─ 2007 2008 cohort 2008─ 2008 2009 cohort (Baseline) Adjusted Group Mean Scores 2007─ 2007 2008 cohort (Baseline) 2009─ 2009 2010 cohort Adjusted Group Mean Scores Adjusted Group Mean Difference (p value) Effect Size

Analyses for Question 5: How did Grade 8 ESOL students perform on MSA reading

Analyses for Question 5: How did Grade 8 ESOL students perform on MSA reading and math? Performed Logistic regressions to compare likelihoods of passing HSA subjects (algebra, English, biology, & government) by ESOL instructional levels for the two graduating classes combined, controlling for background variables Students at Higher ESOL Instructional Level Students at Lower ESOL Instructional Level Odds of Passing an HSA Odds Ratio (Comparing Probabilities of Passing an HSA) P Value and Effect Size

Findings for Question 1: To what extent did secondary ESOL students make annual progress

Findings for Question 1: To what extent did secondary ESOL students make annual progress toward the state target for acquiring English language proficiency? n n Most secondary grades and subgroups made state-required annual progress in English acquisition (percentage of students having a 15 -point or more annual increase in LAS-Links overall scores). Grade 9 and special education students in all grades were most likely to fall below the state target. More lower-level ESOL students made the annual progress in English language proficiency than higher-level ESOL students did. Differences in annual progress among racial/ethnic and service groups became smaller, negligible, none, or reversed across cohorts in most secondary grades.

Findings for Question 2: To what extent did secondary ESOL students attain English language

Findings for Question 2: To what extent did secondary ESOL students attain English language proficiency toward the state target? n n All secondary grades and most subgroups attained the staterequired English proficient level (percentage of students at the advanced level in overall LAS-Links and the proficient or advanced level in all subtests). Special education students in most grades and some racial/ethnic groups in Grade 9 did not meet the state target. More higher-level ESOL students attained the advanced level in English language proficiency than lower-level ESOL students did. White students led other racial/ethnic groups in attaining the target proficient level in most grades

Findings for Question 3: How did Grade 8 ESOL students perform on MAP-R? Yearly

Findings for Question 3: How did Grade 8 ESOL students perform on MAP-R? Yearly achievement changes were found insignificant for Grade 8 MAP-R RIT scores.

Findings for Question 4: How did Grade 8 ESOL students perform on MSA reading

Findings for Question 4: How did Grade 8 ESOL students perform on MSA reading and mathematics? Yearly achievement growth was found significant for MSA reading and mathematics between cohorts 2008 and 2009. Adjusted Means and Mean Differences Comparing 2007– 2008 with 2008– 2009 MSA Outcome Measure Adjusted Means Yearly Achievement 2008– 2009 cohort 2007– 2008 cohort (Baseline) Mean N N Mean Difference Std. Error Mean MSA Reading 377. 25 350 382. 35 347 5. 1* 2. 07 MSA Math 392. 26 483 399. 07 352 6. 80* 2. 58 Note. For both MSA reading and mathematics: p <. 05 and effect size =. 18

Findings for Question 4 (continued): How did Grade 8 ESOL students perform on MSA

Findings for Question 4 (continued): How did Grade 8 ESOL students perform on MSA reading and mathematics? Findings about achievement growth for MSA reading and mathematics were mixed between cohorts 2008 and 2010 Adjusted Means and Mean Differences Comparing 2007– 2008 with 2009– 2010 MSA Adjusted Means Outcome Measure MSA Reading MSA Math 2007– 2008 cohort (Baseline) Yearly Achievement 2009– 2010 cohort Mean Difference Std. Error Mean N 377. 25 350 383. 35 322 6. 29* 2. 37 391. 87 483 389. 53 444 -2. 34 2. 60 Note. For MSA reading: p <. 05 & ES = 0. 20; for MSA mathematics: p >. 05 & ES = -0. 06

Findings for Question 5: Were students receiving higher-level ESOL instruction more likely to pass

Findings for Question 5: Were students receiving higher-level ESOL instruction more likely to pass HSA subjects than those receiving lower-level ESOL instruction? Students who received higher-level ESOL instruction had a significantly higher probability of passing an HSA than their lower-level peers, across all HSA subjects. Odds of Passing HSA Biology by ESOL Instructional Level N HSA Biology Instructional Level 5 Level 4 Level 3 Level 5 Levels 4+3 Odds Ratio 615 373 79 615 452 Effect Size P value 2. 12 0. 000 0. 41 1. 99 0. 009 0. 38 2. 42 0. 000 0. 49