Less Is More An Application of Propensity Score

1. Introduction • Starting point: – Many young children struggle in elementary school –

1. Introduction • Grade retention in Flanders: – Relatively high rate • For example:

1. Introduction – Relatively high rate … especially in Grade 1 § About 7%

1. Introduction • Research question: – Is Grade 1 retention an effective practice or

2. Method • Subjects: representative sample from the Flemish Si. BO-project – 3624 first-graders,

2. Method • Instruments – Psychosocial growth: Teacher questionnaire § rated yearly by the

2. Method • Instruments (continued) – Propensity of repeating Grade 1: § § §

2. Method • Analyses: 4 -steps-procedure – Step 1: identification of ‘true’ confounders of

2. Method • Analyses: 4 -steps-procedure (continued) – Step 3: decile stratification § 10

2. Method • Analyses: 2 comparison strategies – Same-grade approach = comparing retainees with

2. Method Research year 1 (age 7) Research year 2 (age 8) Research year

2. Method • Analyses: 2 comparison strategies (continued) – Why? ~ 2 different questions

3. Results • Propensity scores – based on 52 prior student characteristics Promoted students

3. Results • Propensity score stratification – Cut-offs for strata based on overlap –

3. Results – Within-stratum balance in propensity score = ≠ ≠

3. Results – Within-stratum balance in 97% of the observed pre-retention student, class and

3. Results • Same-grade comparisons – On average: § during their retention year, Grade

3. Results • Same-age comparisons – On average: § Grade 1 repeaters would have

4. Conclusions and discussion • Overall, Grade 1 retainees do not seem to benefit

4. Conclusions and discussion • Future research is needed – Sensitivity analyses – Other

----------------------------------------------------------2 ND BIENNIAL MEETING OF THE EARLI SPECIAL INTEREST GROUP 18 “Educational Effectiveness: Models,

Thank you for your attention! Any suggestions or comments are welcome: mieke. goos@ped. kuleuven.

Slides: 39

Download presentation

Less Is More? An Application of Propensity Score Stratification to First-Grade Retention Mieke Goos, Jan Van Damme, Patrick Onghena and Katja Petry SREE 2010

1. Introduction • Starting point: – Many young children struggle in elementary school – Countries deal with these early problems in a different way – Internationally frequently applied measure = grade retention

1. Introduction • Grade retention in Flanders: – Relatively high rate • For example: PISA 2003

1. Introduction – Relatively high rate … especially in Grade 1 § About 7% of Flemish children repeat Grade 1 – Socially approved by educators, policy makers and parents → being a grade retainee in Flanders has a different connotation than for example in the US (negative overtone) – No formal rules regarding grade promotion (no national/state standardized test procedures) → retention decision = joint decision by teacher and parents

1. Introduction • Research question: – Is Grade 1 retention an effective practice or not? • Focus of this study: – children’s psychosocial growth throughout elementary school

2. Method • Subjects: representative sample from the Flemish Si. BO-project – 3624 first-graders, of which 298 were retained – 222 classes – 121 schools followed until Grade 6

2. Method • Instruments – Psychosocial growth: Teacher questionnaire § rated yearly by the teacher § items on a 1 to 6 point Likert scale § 7 subscales Social skills • Popularity among classmates • Aggressive behavior • Hyperactive behavior • Asocial behavior Dynamic-affective attitudes and skills • Independent participation • School well-being • Self-confidence

2. Method • Instruments (continued) – Propensity of repeating Grade 1: § § § § official records achievement tests Standard Progressive Matrices teacher questionnaire about the child parent questionnaire teacher questionnaire about teacher didactics school staff questionnaire • 68 prior student characteristics • 59 prior class characteristics • 42 prior school characteristics

2. Method • Analyses: 4 -steps-procedure – Step 1: identification of ‘true’ confounders of Grade 1 retention § prior student, class and school characteristics § that are related to both treatment (i. e. , Grade 1 retention) and outcome (i. e. , children’s individual psychosocial growth) – Step 2: estimation of propensity scores based on these confounders § 3 -level logistic regression analysis (students – classes – schools)

2. Method • Analyses: 4 -steps-procedure (continued) – Step 3: decile stratification § 10 strata of equal size – Step 4: estimation of average psychosocial effects § 3 -level curvilinear growth curve analyses (measurements – students – schools)

2. Method • Analyses: 2 comparison strategies – Same-grade approach = comparing retainees with their younger grade-mates – Same-age approach = comparing retainees with their age-mates who were promoted to a higher grade

2. Method Research year 1 (age 7) Research year 2 (age 8) Research year 3 (age 9) 3 rd grade Cohort 1 E Cohort 2 on ti mo (not in Si. BO dataset) o Pr 2 nd grade C n tio o m 1 st grade A o Pr Grade retention on ti mo o B D Grade retention Pr F H on ti mo o Pr Grade retention G

2. Method • Analyses: 2 comparison strategies (continued) – Why? ~ 2 different questions § How do Grade 1 repeaters, at the cost of one extra year of education, develop in comparison to younger children with whom they will eventually finish elementary school? SAME-GRADE APPROACH § How would Grade 1 retainees have developed, had they been promoted to Grade 2 instead? SAME-AGE APPROACH

3. Results • Propensity scores – based on 52 prior student characteristics Promoted students M = -4. 47 Retained students M = -0. 12

3. Results • Propensity score stratification – Cut-offs for strata based on overlap – Division into 10 strata of equal size

3. Results – Within-stratum balance in propensity score = ≠ ≠

3. Results – Within-stratum balance in 97% of the observed pre-retention student, class and school characteristics → Retained and promoted children within a certain stratum are equivalent (within sampling fluctuations) in terms of risk factors preceding retention

3. Results • Same-grade comparisons – On average: § during their retention year, Grade 1 retainees show a similar psychosocial functioning in comparison to younger grademates who are at similar risk of being retained § but … over time they (mostly) grow significantly slower they end up showing more hyperactive behavior, feeling less well at school etc. ! One exception: popularity among classmates

3. Results

3. Results sign

3. Results

3. Results sign

3. Results ! One exception ! sign

3. Results • Same-age comparisons – On average: § Grade 1 repeaters would have developed a similar or even better psychosocial functioning, had they been promoted to Grade 2 instead, both in the short and long run

3. Results sign

3. Results

3. Results sign

3. Results

3. Results sign

4. Conclusions and discussion • Overall, Grade 1 retainees do not seem to benefit much from their retention year – over time they grow slower compared to grademates, making them end up feeling less well at school etc. at the end of elementary school – while they would have developed a similar or even better psychosocial functioning, had they been promoted to Grade 2 instead → Practical implication: Our results call the practice of Grade 1 retention in Flanders into question

4. Conclusions and discussion • Future research is needed – Sensitivity analyses – Other outcome: growth in math and reading skills – Moderating effects: provision of additional support

----------------------------------------------------------2 ND BIENNIAL MEETING OF THE EARLI SPECIAL INTEREST GROUP 18 “Educational Effectiveness: Models, Methods and Applications” Leuven, Belgium 25 -27 August 2010 http: //www. sigee 2010. org ---------------------------------------------------------- Keynote lectures by Prof. Dr. Stephen Raudenbush, Prof. Dr. Robert E. Slavin, Prof. Dr. Jürgen Baumert and Prof. Dr. Jan-Eric Gustafsson

Thank you for your attention! Any suggestions or comments are welcome: mieke. goos@ped. kuleuven. be