The Role of Optimal Challenge in Adaptive Elearning

The Role of Optimal Challenge in Adaptive Elearning: Evidence from Field Experiments with Middle School Students De Liu Information and Decision Sciences Carlson School of Management University of Minnesota Joint work with Tao Li, Sean Xin Xu | Oct 17, 2019 @ DAE, Knoxville, TN 1

E-Learning provides many opportunities for online experimentation • Rise of online and mobile learning has redefined learning • These platforms are incredible fertile playgrounds for design and experimentation. – Big data (volume + granularity), real-time personalized intervention – Applications of ML/AI tools, new challenge issues (disengagement, selfselection, fairness) 2

A research stream aimed to help middle school students learn outside of the classrooms using online learning platforms. • Collaborate with a few middle school in Beijing • Currently focus on learning English during summer/winter breaks • Online learning platform + machine learning + learning/engagement theories Problem Sequencing Digital Learning Divide Interleaving vs blocking designs Self-selection & polarization Adaptive challenge design Re-engaging underachievers 3

How to choose a problem with the right challenge level for learners? • Large question banks exist for online learning • Within each topic domain, we can choose from many practice problems with different challenge levels • The question is how to choose? Challenge Level Strong learner Weak learner Challenge consistency 4

Agenda • Introduction & motivation • Related literature • Research hypotheses • Experiment design • Results • Discussion 5

Relationship with Personalized Learning • Personalized learning is pivotal for e-learning designs Personaliz ed learning Topic Gap Detection Adaptive Challenge Design Indispensable but neglected 6

Relationship with Adaptive Learning • Adaptively accommodate individual’s heterogeneities and learning progress (Schneps et al. 2010) Macro adaptive: adapt to different learning goals Self-select learning goal Aptitude adaptive: Adapt to learner types e. g. different learning plans for different learning styles This research Micro adaptive: adapt to learning dynamics e. g. , adapt learning based on learner progress 7

Prior Research on Adaptive Challenge • Maintain a medium challenge level – Endler et al. (2012) adjust problem difficulty to maintain a 50% success rate in the last two tasks – Sampayo-Vargas et al. (2013) adjust the challenge to stay within a (unspecified) success rate band. • Item Response Theory (Wauters et al. 2010) – Item challenge match learner’s ability 8

Some gaps in existing online learning research • Theoretical ambiguity – What is the optimal level of challenge for learners with different aptitudes? – Is it because the level of challenge is good or simply that the challenge level is consistent? – Field experiment! • Implementation sophistication – Existing research is very underdeveloped in real-time estimation of learner's mastery levels in specific areas and they evolve – Machine learning can help! 9

Agenda • Introduction & motivation • Related Literature • Research hypotheses • Experiment design • Results • Discussion 10

Optimal Challenge Level - Flow theory • Match between perceived challenge and perceived ability (Csikszentmihalyi 1975) • Optimal challenge level is different for different individuals – Low ability Low challenge (flow) – High ability High challenge (flow) • H 1: For weak learners: Low challenge > High challenge (in learning performance) • H 2: For strong learners, High challenge > Low challenge (in learning performance) 11

Sensitivity to Challenge Levels • Differences between flow and boredom are not found in some scenarios. • Replace the boredom to relaxation (Csikszentmihalyi 1997; Engeser and Rheinberg 2008) • H 3: Learning performance disparity between low and high challenges is smaller for strong learners than for weak learners. Csikszentmihalyi (1997) 12

Does Challenge Consistency Matter? Challenge Level (mean) Challenge consistency (variance) • H 4: Challenge consistency Better performance Fluctuations in challenges Unexpected impediment reassessment or abandon (Butler and Winne 1995) + Negative attribution framing (Weiner 1985) 13

Agenda • Introduction & motivation • Related literature • Research hypotheses • Experiment design • Results • Discussion 14

Research Context • 756 students (7 -th grade) from 6 middle schools in Beijing • English reading comprehension • Winter break reading assignments (Jan. ~ Feb. 2019) – 7 English articles per week, each followed by 3 -5 questions. – Students are expected to complete them, but no penalty if they do not (other than scolded by teachers) – 13, 076 completed student-article assignments, 86. 7% completion rates 15

System User Interface 16

Experiment Design Random Challenge Inconsistent Consistent Low Challenge High Challenge • Dependent variable – Posttest accuracy (correctness): Number of correct answers / number of questions. • Conditions – Control: randomly choose problems from a pool of candidates – High Challenge: problems ranked among 20% lowest predicted accuracy for this learner at the present time – Low Challenge: problems ranked among 20% highest predicted accuracy for this learner at the present time • Learner aptitude: – Strong learner: pre-test final exam score above the median score in one's class – Weak learner: pre-test final exam score below the median score in one's class 17

Experimental Procedure • Between subject: The assigned condition is fixed during the study period. 4 weeks 18

Accuracy Prediction • Our assignment is based on dynamic, personalized accuracy prediction – Predict accuracy for student i on English article j at day t. • Heterogeneous ensemble with 5 -fold cross-validation – SVM + BP Neural Network + Ridge Regression + Ordered Logistic Regression • Predictive performance (ex post) – Accuracy (% of questions answered correctly)：R-square = 58. 6% – Number of correct answers per article: 66. 5% accurate. • 97% within 1 deviation from the actual # of correct answers. 19

Features used for accuracy prediction 20

Knowledge gap detection • Use to select a subset of candidate problems targeted at a certain knowledge (topic) gap. • Hidden Markov Model + manually coded problem-to-topic mapping – Portray the evolvement of topic proficiency – Higher proficiency more likely to correctly answer the problems related to the topic. 21

Randomization check • No significant difference in the pre-test final exam scores 22

Agenda • Introduction & motivation • Related literature • Research hypotheses • Experiment design • Results • Discussion 23

Posttest Learning Performance Comparisons Random • H 1 (√)：Weak learner, low challenge > high challenge • H 2 (X)：Strong learner, high challenge > low challenge • H 3 (√)：Disparity between high and low challenge is lower for strong learners • H 4 (√)：Consistent challenge better performance* Random High Low 24

Regression results H 1 is supported H 1: For weak learners, Low > High challenge 25

Regression results H 3 is supported H 3: For Strong learner, disparity is smaller 26

Regression results H 2 is not supported (sum of the two coefficients) H 2: For strong learners, High > Low challenge 27

Regression results H 4 is supported Challenge consistency Better performance 28

Agenda • Introduction & motivation • Related literature • Research hypotheses • Experiment design • Results • Discussion 29

Challenges in Design and Analysis • Treatment based on predicted accuracy is inherently imprecise We removed the cases that are too "wrong" - Predicted accuracy > 75% for High challenge - Predicted accuracy < 65% for low challenge How to analyze such a data set based on imprecise treatments? 30

Challenges in Design and Analysis (2) • The experiment design isn't ideal for teasing apart level vs. consistency effects – The control group differs in both consistency and level! • Can we construct synthetic datasets ex post to address this design flaw? 31

Challenges in Design and Analysis (3) • Self-selection issue exists (and may cause biases) • In another repetition of the experiment (where there is less teacher intervention), only 38% of students completed their assignments – In this one, 86. 7% completed their assignments. 32

Thank You! De Liu deliu@umn. edu

Forecast Accuracy • Correctness：R-square 58. 6% • Number of correct answered problems in each exercise – Accuracy: 66. 5% ; 97% within 1 problem forecast error 34

– Consistent result after the following manipulation adjustment – Conflicts: Predicted correctness > 0. 75 for High challenge ； Predicted correctness < 0. 65 for low challenge – Delete # of conflicts more than 1 times. – Delete 56 individuals including 27 in high challenge and 29 in low challenge 35

Manipulation Check 36

Robustness Check – Different definitions of strong/weak learners – Main analysis we use top-50% as strong learners, here we try different %. 37