The Effectiveness of Khan Academys SAT Preparation Program

The Effectiveness of Khan Academy’s SAT Preparation Program by Ted Wardell June 13 th, 2019 Committee: Dr. Judith Bazler, Chair Dean John Henning Dr. Vecihi Zambak (Dr. Letitia Graybill)

Introduction ● Admissions to college is competitive with potential inequities both in terms of acceptance as well as scholarship opportunities. ● A large component to that admission process is some form of standardized test score such as SAT, ACT, or individual college entrance exams. ● Most, if not all, of such exams has a mathematical component, highlighting the importance of mathematical achievement throughout one’s schooling. 2

Introduction ● The manner in which students prepare for these standardized tests, in particular the mathematical section varies considerably. ● Some students rely solely on what was learned in school or use only open educational resources (OER). ● Others pay for preparatory tools such as books, digital programs, coursework, or private tutoring. 3

Introduction ● The PSAT/SAT suite of exams is one of the most common of these exams and one of the largest drivers of the test preparation market. In fact, 2018 saw a record number of SAT takers (2, 136, 539), up 25% from 2017. ● The aforementioned variability in both preparation offerings and costs thereof creates a potential inequity in the competitive college admissions process. 4

Introduction Average SAT Math Score by Family Income Year Family income less than $20, 000 Family income more than $200, 000 2016 453 586 2015 455 587 2014 459 588 2013 462 586 2012 461 589 5

Introduction ● Khan Academy is one of the most, if not the most, prevalent providers of an OER in the world. ● In partnership with the College Board, Khan Academy is attempting to address the inequity in test prep with its development and 2016 release of a free SAT preparation program. ● The purpose of this quantitative study was to add to the body of research regarding the effectiveness of Khan Academy’s SAT Mathematics Preparation Program. 6

The argument goes… ● People with college degrees experience significant advantages over those without in terms of finances, health & wellness, and general life satisfaction (Julian and Kominski, 2011; U. S Census, 2015; College Board, 2017; Baum, Ma & Pays, 2010; Ma, Pender & Welch, 2016; Rostron, Boies & Arias, 2010). ● Access is decreasing; the number of seats available at all colleges and universities has been greatly outpaced by the number of students seeking admission for nearly 40 years (Bound, 2009). 7

The argument goes… ● High School students and their families are doing more than ever to improve access such as taking increasingly rigorous course loads, carefully selecting a broader variety of extracurricular activities, or hiring professional coaches or consultants (College Board, 2016; Hoover, 2016; Greisemer 2015) ● Students from socioeconomically disadvantaged backgrounds remain less likely to enter or complete higher education (Gamoran 2001; Karen 2002; Roksa et al. 2007) 8

PSAT/SAT ● Regardless of any other factors in the college application process, the SAT remains a significant part of the path to most colleges. ● Research suggest more than 82% of colleges place high to moderate importance on such scores (Clinedinst & Koranteng, 2017) ● “Achieving a high SAT score is very much a part of the college admissions equation. Although high scores by no means guarantee admission to selective schools, low scores very often disqualify students from admission” (Buchanan, 2010). 9

Test Preparation ● A thriving industry has emerged to assist students in this process and now makes available an unprecedented array of SAT preparatory options—books, flashcards, online programs, one-on-one tutoring, and large- and small-group programs (Lilly and Montgomery 2011). ● Today the company, Kaplan, its main counterpart, the Princeton Review, and many others, including private tutors, are all part of a $4. 5 -billion-a-year industry catering largely to the well-to-do in America who feel that the test can be gamed (Balf, 2014). ● “The very existence of the private test-prep industry suggests a student's educational achievement may be directly related to their income bracket” (Turner, 2014). 10

Khan Academy ● In 2016, College Board released its redesigned SAT while partnering with Khan Academy to produce a free high quality test prep program. ● With a motto of “Our mission is to provide a free, world‑ class education for anyone, anywhere”, Khan Academy claims to offer “a personalized learning resource for all ages”. ● However, limited research into the website’s effectiveness exists. The purpose of this study was to add the body of research by investigating the effectiveness of Khan Academy’s SAT math preparation program. 11

Research Questions 1. Do users of Khan Academy’s SAT Math program perform better on the math section of the PSAT that non-users? 2. Is there a correlation between the amount of time a student spends using Khan Academy’s SAT Math program and PSAT math scores of high school students? 3. Is there a correlation between the number of total problems or program-recommended problems a student completes in Khan Academy’s SAT Math program and PSAT math scores of high school students? 4. Is the use of Khan Academy’s Mathematics SAT preparation program predictive of PSAT math scores of high school students when controlling for differences in demographics such as grade level, gender, socioeconomic level, ethnicity, or students with disabilities? 12

Methodology ● The study took place at Monmouth Regional High School in Tinton Falls starting with approximately 450 students of diverse backgrounds. ● In the spring of 2018, Khan Academy’s SAT Math program was introduced into all math courses containing students eligible for the October 2018 PSAT. ● Students were encouraged to use the program through the spring, summer, and fall leading up to the October PSAT with suggested intervals of use. ● In the fall of 2018, 309 students sat for October PSAT with scores arriving in mid-December. 13

Data Analysis, RQ 1 ● Research question 1 was analyzed in several methods. ● First, an independent samples t-test was used to compare the means of 2018 PSAT math scores of the students who used Khan Academy, the experimental group, and those that did not, the control group. ● Additionally, another independent samples t-test was used to compare the mean gain score of the students who used Khan Academy and those who did not. (A gain score is defined as the net change in PSAT math scores from 2017 to 2018 for those students who took the exam in both years. ) 14

Data Analysis, RQ 1 continued. . ● In subsequent analyses, the entire group of 2018 PSAT test-takers was viewed as the experimental group while the groups of PSAT test-takers from 2015 – 2017 were viewed as the control groups. ● For these analyses, a one-way ANOVA was used to compare the means of each of the 4 years of PSAT math scores (2015 -2018) as four individual groups. ● Furthermore, a post hoc Tukey’s Honest Significance Difference was used with each of the 4 groups of PSAT math scores from 2015 to 2018 to further investigate differences in means between all possible pairs amongst the 4 groups. 15

Data Analysis, RQ 2 & 3 ● Since research questions 2 and 3 both sought to investigate for correlations, these analyses were conducted in tandem. ● Research questions 2 and 3 were analyzed using a Pearson multiple regression between the PSAT Math scores against hours and number of overall questions and number of recommended questions practiced in the Khan Academy Math SAT program 16

Data Analysis, RQ 4 ● Research question 4 was analyzed using hierarchical multiple regression on the PSAT Math scores allowing significant demographic and academic predictors to enter the equation in a stepwise fashion. ● Including use of Khan Academy, this procedure considered nine potential variables for entry, including IEP status (yes vs. no), gender (male vs. female), Asian race (yes vs. no), Black race (yes vs. no), Latino race (yes vs. no), White race (yes vs. no), grade level (10 th vs. 11 th or 12 th grade), lunch status (no lunch vs. reduced cost or free lunch) and age (15, 16, 17+). 17

Findings, RQ 1 ● The general findings suggest that use of Khan Academy’s SAT Math program does improve student outcomes on the math section of the PSAT. ● Students who used the Khan Academy SAT Math program for at least an hour performed better on 2018 PSAT Math compared to those who didn't use the program or used the program for less than an hour (t (307) = 6. 71, p <. 001) ● For the 192 students who took the PSAT Math in 2017 and 2018, a statistically significant gain was not found (t (192) =. 54, p =. 541) 18

Findings, RQ 1 continued… ● The prior statistical measures focused solely on the scores of the 2018 PSAT takers grouped using a post-determined level of use of Khan Academy. ● As an alternative analysis, a one-way ANOVA was used to compare the means of each of the 4 years of PSAT math scores (2015 -2018) as four individual groups with results that demonstrated significant differences in means existed (F = 5. 085, p =. 002) 19

Findings, RQ 1 continued… ● Additionally, a Tukey’s honest significant difference (HSD) was used post hoc to evaluate where the differences in means exist between the four groups of PSAT test-takers. ● This is important because Tukey’s HSD can demonstrate that the 2018 results differ significantly from each of the 3 previous years while confirming that none of those 3 years differed significantly from each other. 20

21

Findings, RQ 2 & 3 ● The results also suggest that only the amount if time the program is used correlates with student PSAT Math scores. Neither the number of total problems completed nor the number of recommended problems completed were correlated with PSAT Math scores. ● Time Practiced (hours) showed to be a weak predictor of 2018 PSAT score (r =. 278). However neither of the variables, Practice Questions or Recommended Questions, predicts 2018 PSAT score (r =. 040 and. 003). All three results were at a significant level (p <. 000). 22

Findings, RQ 4 ● Lastly, the hierarchal multiple regression analysis suggests that a handful demographic factors influenced PSAT math scores, however, controlling for those factors reveals that use of Khan Academy’s SAT program remains predictive of PSAT math scores. ● The results indicated that IEP status and use of the Khan Academy program were the strongest predictors in the model (IEP β = -0. 348, t = -7. 04, p <. 001 and use of Khan Academy β = 0. 221, t = 4. 15, p <. 001 respectively) 23

Big Take-aways ● the most noteworthy finding might be that Khan Academy’s SAT Math program appears to have influenced the PSAT Math scores of its users in a positive direction. ● The type of use did not seem to matter, only the amount of time the program was used. ● This suggests that simply the introduction and encouraged use of Khan Academy’s SAT Math program can elicited positive results on the math section of PSAT/SAT suite of exams. 24

Limitations ● ● Quasi-experimental designs are not as rigorous as traditional experimental. Higher risk for selection bias in post-determined control & experimental groups. Single school study lacks generalizability. Population fluidity & lack of control over test-takers 25

Further study ● Traditional experimental design expanded to multiple high schools. ● More controlled use, i. e. follow the programs lead, minimum # of problems, minimum amount of time, etc… ● Comparability to other preparation mechanisms, i. e. prep book, in-person course, private tutoring, etc… ● Student driven vs. teacher driven ● Student perception (qualitative or mixed methods) 26

Tying it all together ● SAT scores remain an integral component to college access. ● Well-established differences exist in SAT scores of the lowest and highest socio -economic brackets. ● A potential contributing factor could be access to “for pay” preparation mechanism, i. e. prep books, software, courseware, private tutoring, etc… ● Khan Academy partnered with College Board in developing its powerful, yet free, SAT preparation offering. ● Both this study and those of Khan Academy in 2017 suggest that using the SAT program in preparation of one of the PSAT/SAT suite of exams results in greater scores on said exams. ● “For free. For everyone. Forever. ” –Khan Academy 27