Bayesian Knowledge Tracing and Other Predictive Models in

Bayesian Knowledge Tracing and Other Predictive Models in Educational Data Mining Zachary A. Pardos PSLC Summer School 2011 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Outline of Talk • Introduction to Knowledge Tracing – History Intuition Model – Demo – – • Random Forests – – • Variations (and other models) Evaluations (baker work / kdd) Description Evaluations (kdd) Time left? – Vote on next topic 2 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing History • Introduced in 1995 (Corbett & Anderson, UMUAI) • Basked on ACT-R theory of skill knowledge (Anderson 1993) • Computations based on a variation of Bayesian calculations proposed in 1972 (Atkinson) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Intuition • Based on the idea that practice on a skill leads to mastery of that skill • Has four parameters used to describe student performance • Relies on a KC model • Tracks student knowledge over time Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing For some Skill K: Given a student’s response sequence 1 to n, predict n+1 1 …. n n+1 0 0 1 1 ? Chronological response sequence for student Y [ 0 = Incorrect response 1 = Correct response] Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Track knowledge over time (model of learning) 0 0 Zach Pardos 1 0 1 Bayesian Knowledge Tracing & Other Models 1 1 PLSC Summer School 2011

Intro to Knowledge Tracing (KT) can be represented as a simple HMM Latent Observed Node representations K = Knowledge node Q = Question node UMAP 2011 Zach Pardos Node states K = Two state (0 or 1) Q = Two state (0 or 1) 7 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Four parameters of the KT model: P(L 0) P(G) P(T) P(L 0) = Probability of initial knowledge P(T) = Probability of learning P(G) = Probability of guess P(S) = Probability of slip P(T) P(G) P(S) Probability of forgetting assumed to be zero (fixed) UMAP 2011 Zach Pardos 8 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Formulas for inference and prediction • Derivation (Reye, JAIED 2004): • Formulas use Bayes Theorem to make inferences about latent variable Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Model Training: Model Training Step - Values of parameters P(T), P(G), P(S) & P(L 0 ) used to predict student responses • Ad-hoc values could be used but will likely not be the best fitting • Goal: find a set of values for the parameters that minimizes prediction error Student A 0 Student B 1 Student C 0 Zach Pardos 0 1 0 1 0 Bayesian Knowledge Tracing & Other Models 1 1 0 PLSC Summer School 2011

Intro to Knowledge Tracing Model Prediction: Model Tracing Step – Skill: Subtraction P(K) 10% 45% 75% 79% 83% Latent (knowledge) Observable (responses) 0 1 1 Student’s last three responses to Subtraction questions (in the Unit) Zach Pardos Bayesian Knowledge Tracing & Other Models P(Q) 71% 74% Test set questions PLSC Summer School 2011

Intro to Knowledge Tracing Influence of parameter values Estimate of knowledge for student with response sequence: 0 1 1 1 1 1 Student reached 95% probability of knowledge After 4 th opportunity P(L 0): 0. 50 P(T): 0. 20 P(G): 0. 14 P(S): 0. 09 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Influence of parameter values Estimate of knowledge for student with response sequence: 0 1 1 1 1 1 Student reached 95% probability of knowledge After 8 th opportunity P(L 0): 0. 50 P(T): 0. 20 P(G): 0. 14 P(S): 0. 09 P(L 0): 0. 50 P(T): 0. 20 P(G): 0. 64 P(S): 0. 03 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing ( Demo ) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Variations on Knowledge Tracing (and other models) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach Do all students enter a lesson with the same background knowledge? P(L 0|S) Observed Node representations K = Knowledge node Q = Question node S = Student node Zach Pardos Node states K = Two state (0 or 1) Q = Two state (0 or 1) S = Multi state (1 to N) Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach Conditional Probability Table of Student node and Individualized Prior node P(L 0|S) CPT of Student node S value P(S=value) 1 1/N 2 1/N 3 1/N … 1/N S value can represent a cluster or type of student instead of ID Zach Pardos • CPT of observed student node is fixed • Possible to have S value for every student ID • Raises initialization issue (where do these prior values come from? ) Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach Conditional Probability Table of Student node and Individualized Prior node CPT of Individualized Prior node • Individualized L 0 values S value P(L 0|S) need to be seeded • This CPT can be fixed or 1 0. 05 the values can be learned 2 0. 30 • Fixing this CPT and 3 0. 95 seeding it with values based on a student’s first … … response can be an N 0. 92 effective strategy This model, that only individualizes L 0, the Prior Per Student (PPS) model Zach Pardos Bayesian Knowledge Tracing & Other Models P(L 0|S) PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach Conditional Probability Table of Student node and Individualized Prior node P(L 0|S) CPT of Individualized Prior node • Bootstrapping prior • If a student answers incorrectly on the first question, she gets a low prior • If a student answers correctly on the first question, she gets a higher prior Zach Pardos S value P(L 0|S) 0 0. 05 1 0. 30 1 1 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach What values to use for the two priors? P(L 0|S) CPT of Individualized Prior node What values to use for the two priors? S value P(L 0|S) 0 0. 05 1 0. 30 1 1 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach What values to use for the two priors? P(L 0|S) CPT of Individualized Prior node 1. Use ad-hoc values S value P(L 0|S) 0 0. 10 1 0. 85 1 1 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach What values to use for the two priors? P(L 0|S) CPT of Individualized Prior node 1. Use ad-hoc values 2. Learn the values S value P(L 0|S) 0 EM 1 1 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach What values to use for the two priors? P(L 0|S) CPT of Individualized Prior node 1. Use ad-hoc values 2. Learn the values 3. Link with the guess/slip CPT S value P(L 0|S) 0 Slip 1 1 -Guess 1 1 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Prior Individualization Approach What values to use for the two priors? P(L 0|S) CPT of Individualized Prior node 1. Use ad-hoc values 2. Learn the values 3. Link with the guess/slip CPT S value P(L 0|S) 0 Slip 1 1 -Guess 1 1 With ASSISTments, PPS (ad-hoc) achieved an R 2 of 0. 301 (0. 176 with KT) (Pardos & Heffernan, UMAP 2010) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Variations on Knowledge Tracing (and other models) UMAP 2011 Zach Pardos 25 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 1. BKT-BF Learns values for these parameters by performing a grid search (0. 01 granularity) and chooses the set of parameters with the best squared error P(L 0) P(T) P(G) P(S) P(G) P(L 0) = Probability of initial knowledge P(T) = Probability of learning P(G) = Probability of guess P(S) = Probability of slip P(T) P(S) . . . P(G) P(S) (Baker et al. , 2010) UMAP 2011 Zach Pardos 26 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 2. BKT-EM Learns values for these parameters with Expectation Maximization (EM). Maximizes the log likelihood fit to the data P(L 0) P(T) P(G) P(S) P(G) P(L 0) = Probability of initial knowledge P(T) = Probability of learning P(G) = Probability of guess P(S) = Probability of slip P(T) P(S) . . . P(G) P(S) (Chang et al. , 2006) UMAP 2011 Zach Pardos 27 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 3. BKT-CGS Guess and slip parameters are assessed contextually using a regression on features generated from student performance in the tutor P(L 0) P(T) P(G) P(S) P(L 0) = Probability of initial knowledge P(T) = Probability of learning P(G) = Probability of guess P(S) = Probability of slip P(T) . . . P(G) P(S) (Baker, Corbett, & Aleven, 2008) UMAP 2011 Zach Pardos 28 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 4. BKT-CSlip Uses the student’s averaged contextual Slip P(L ) = Probability of initial knowledge 0 parameter learned across all incorrect actions. P(T) = Probability of learning P(G) = Probability of guess P(S) = Probability of slip P(L 0) P(T) . . . P(G) P(S) (Baker, Corbett, & Aleven, 2008) UMAP 2011 Zach Pardos 29 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 5. BKT-Less. Data Limits students response sequence length to P(L ) = Probability of initial knowledge 0 the most recent 15 during EM training. P(T) = Probability of learning P(G) = Probability of guess P(S) = Probability of slip P(L 0) P(T) . . . P(G) P(S) Most recent 15 responses used (max) (Nooraiei et al, 2011) UMAP 2011 Zach Pardos 30 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 6. BKT-PPS Prior per student (PPS) model which individualizes the prior parameter. Students are assigned a prior based on their response to the first question. P(L 0|S) 0) P(T) P(L 0) = Probability of initial knowledge P(T) = Probability of learning P(G) = Probability of guess P(S) = Probability of slip P(T) . . . Observed P(G) P(S) (Pardos & Heffernan, 2010) UMAP 2011 Zach Pardos 31 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 7. CFAR Correct on First Attempt Rate (CFAR) calculates the student’s percent correct on the current skill up until the question being predicted. Student responses for Skill X: 0 1 0 1 _ Predicted next response would be 0. 50 (Yu et al. , 2010) UMAP 2011 Zach Pardos 32 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 8. Tabling Uses the student’s response sequence (max length 3) to predict the next response by looking up the average next response among student with the same sequence in the training set Training set Student A: 0 1 1 0 Student B: 0 1 1 1 Student C: 0 1 1 1 Max table length set to 3: Table size was 20+21+22+23=15 Test set student: 0 0 1 _ Predicted next response would be 0. 66 (Wang et al. , 2011) UMAP 2011 Zach Pardos 33 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing 9. PFA Performance Factors Analysis (PFA). Logistic regression model which elaborates on the Rasch IRT model. Predicts performance based on the count of student’s prior failures and successes on the current skill. (Pavlik et al. , 2009) UMAP 2011 Zach Pardos 34 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Methodology Dataset Study Evaluation • Cognitive Tutor for Genetics – 76 CMU undergraduate students – 9 Skills (no multi-skill steps) – 23, 706 problem solving attempts – 11, 582 problem steps in the tutor – 152 average problem steps completed per student (SD=50) – Pre and post-tests were administered with this assignment Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Methodology Study model in-tutor prediction • Predictions were made by the 9 models using a 5 fold cross-validation by student -B T K Student 1 • F EM KT … B B Actual Skill A Resp 1 0. 10 0. 22 0 Skill A Resp 2 …. 0. 51 0. 26 1 Skill A Resp N 0. 77 0. 40 1 Skill B Resp 1 … 0. 55 0. 60 1 Skill B Resp N 0. 41 0. 61 0 Accuracy was calculated with A’ for each student. Those values were then averaged across students to report the model’s A’ (higher is better) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Results Study in-tutor model prediction A’ results averaged across students Model A’ Zach Pardos BKT-PPS 0. 7029 BKT-BF 0. 6969 BKT-EM 0. 6957 BKT-Less. Data 0. 6839 PFA 0. 6629 Tabling 0. 6476 BKT-CSlip 0. 6149 CFAR 0. 5705 BKT-CGS 0. 4857 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Results Study in-tutor model prediction No significant differences within these BKT A’ results averaged across students Model A’ BKT-PPS 0. 7029 BKT-BF 0. 6969 BKT-EM 0. 6957 Significant BKT-Less. Data differences between these BKT PFA and PFA Tabling Zach Pardos 0. 6839 0. 6629 0. 6476 BKT-CSlip 0. 6149 CFAR 0. 5705 BKT-CGS 0. 4857 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Methodology Study ensemble in-tutor prediction • 5 ensemble methods were used, trained with the same 5 fold cross-validation folds features Student 1 • -B T K F EM KT … B B label Actual Skill A Resp 1 0. 10 0. 22 0 Skill A Resp 2 …. 0. 51 0. 26 1 Skill A Resp N 0. 77 0. 40 1 Skill B Resp 1 … 0. 55 0. 60 1 Skill B Resp N 0. 41 0. 61 0 Ensemble methods were trained using the 9 model predictions as the features and the actual response as the label. Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Methodology Study ensemble in-tutor prediction • Ensemble methods used: 1. 2. 3. 4. 5. Linear regression with no feature selection (predictions bounded between {0, 1}) Linear regression with feature selection (stepwise regression) Linear regression with only BKT-PPS & BKT-EM Linear regression with only BKT-PPS, BKT-EM & BKT-CSlip Logistic regression Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Results Study in-tutor ensemble prediction A’ results averaged across students Model A’ Ensemble: Lin. Reg with BKT-PPS, BKT-EM & BKT-CSlip 0. 7028 Ensemble: Lin. Reg with BKT-PPS & BKT-EM 0. 6973 Ensemble: Lin. Reg without feature selection 0. 6945 Ensemble: Lin. Reg with feature selection (stepwise) 0. 6954 Ensemble: Logistic without feature selection 0. 6854 No significant difference between ensembles Ta Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Results Study in-tutor ensemble & model prediction A’ results averaged across students Model A’ BKT-PPS 0. 7029 Ensemble: Lin. Reg with BKT-PPS, BKT-EM & BKT-CSlip 0. 7028 Ensemble: Lin. Reg with BKT-PPS & BKT-EM 0. 6973 BKT-BF 0. 6969 BKT-EM 0. 6957 Ensemble: Lin. Reg without feature selection 0. 6945 Ensemble: Lin. Reg with feature selection (stepwise) 0. 6954 Ensemble: Logistic without feature selection 0. 6854 BKT-Less. Data 0. 6839 PFA 0. 6629 Tabling 0. 6476 BKT-CSlip 0. 6149 CFAR 0. 5705 BKT-CGS 0. 4857 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Results Study in-tutor ensemble & model prediction A’ results calculated across all actions Model A’ Ensemble: Lin. Reg with BKT-PPS, BKT-EM & BKT-CSlip Ensemble: Lin. Reg without feature selection Ensemble: Lin. Reg with feature selection (stepwise) Ensemble: Logistic regression without feature selection Ensemble: Lin. Reg with BKT-PPS & BKT-EM BKT-BF BKT-PPS PFA BKT-Less. Data CFAR Tabling Contextual Slip BKT-CGS Zach Pardos Bayesian Knowledge Tracing & Other Models 0. 7451 0. 7428 0. 7423 0. 7359 0. 7348 0. 7330 0. 7310 0. 7277 0. 7220 0. 6723 0. 6712 0. 6396 0. 4917 PLSC Summer School 2011

Random Forests In the KDD Cup • Motivation for trying non KT approach: – Bayesian method only uses KC, opportunity count and student as features. Much information is left unutilized. Another machine learning method is required • Strategy: – Engineer additional features from the dataset and use Random Forests to train a model Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests • Strategy: – Create rich feature datasets that include features created from features not included in the test set Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests • Created by Leo Breiman • The method trains T number of separate decision tree classifiers (50 -800) • Each decision tree selects a random 1/P portion of the available features (1/3) • The tree is grown until there at least M observations in the leaf (1 -100) • When classifying unseen data, each tree votes on the class. The popular vote wins or an average of the votes (for regression) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests Feature Importance Features extracted from training set: • Student progress features (avg. importance: 1. 67) – – Number of data points [today, since the start of unit] Number of correct responses out of the last [3, 5, 10] Zscore sum for step duration, hint requests, incorrects Skill specific version of all these features • Percent correct features (avg. importance: 1. 60) – % correct of unit, section, problem and step and total for each skill and also for each student (10 features) • Student Modeling Approach features (avg. importance: 1. 32) – The predicted probability of correct for the test row – The number of data points used in training the parameters – The final EM log likelihood fit of the parameters / data points Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests • Features of the user were more important in Bridge to Algebra than Algebra • Student progress features / gaming the system (Baker et al. , UMUAI 2008) were important in both datasets Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests Algebra Rank Feature set RMSE Coverage 1 All features 0. 2762 87% 2 Percent correct+ 0. 2824 96% 3 All features (fill) 0. 2847 97% Bridge to Algebra Rank Feature set RMSE Coverage 1 All features 0. 2712 92% 2 All features (fill) 0. 2791 99% 3 Percent correct+ 0. 2800 98% Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests Algebra Rank Feature set RMSE Coverage 1 All features 0. 2762 87% 2 Percent correct+ 0. 2824 96% 3 All features (fill) 0. 2847 97% Bridge to Algebra Rank Feature set RMSE Coverage 1 All features 0. 2712 92% 2 All features (fill) 0. 2791 99% 3 Percent correct+ 0. 2800 98% • Best Bridge to Algebra RMSE on the Leaderboard was 0. 2777 • Random Forest RMSE of 0. 2712 here is exceptional Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests Algebra Rank Feature set RMSE Coverage 1 All features 0. 2762 87% 2 Percent correct+ 0. 2824 96% 3 All features (fill) 0. 2847 97% Bridge to Algebra Rank Feature set RMSE Coverage 1 All features 0. 2712 92% 2 All features (fill) 0. 2791 99% 3 Percent correct+ 0. 2800 98% • Skill data for a student was not always available for each test row • Because of this many skill related feature sets only had 92% coverage Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Conclusion from KDD • Combining user features with skill features was very powerful in both modeling and classification approaches • Model tracing based predictions performed formidably against pure machine learning techniques • Random Forests also performed very well on this educational data set compared to other approaches such as Neural Networks and SVMs. This method could significantly boost accuracy in other EDM datasets. Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Random Forests Hardware/Software • Software – MATLAB used for all analysis • Bayes Net Toolbox for Bayesian Networks Models • Statistics Toolbox for Random Forests classifier – Perl used for pre-processing • Hardware – Two rocks clusters used for skill model training • 178 CPUs in total. Training of KT models took ~48 hours when utilizing all CPUs. – Two 32 gig RAM systems for Random Forests • RF models took ~16 hours to train with 800 trees Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Time left? Choose the next topic • • • KT: 1 -35 Prediction: 36 -67 Evaluation: 47 -77 sig tests: 69 -77 Regression/sig tests: 80 -112 Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Individualize Everything? UMAP 2011 Zach Pardos 55 Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Fully Individualized Model (Pardos & Heffernan, JMLR 2011) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Fully Individualized Model S identifies the student (Pardos & Heffernan, JMLR 2011) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Fully Individualized Model T contains the CPT lookup table of individual student learn rates (Pardos & Heffernan, JMLR 2011) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing Fully Individualized Model P(T) is trained for each skill which gives a learn rate for: P(T|T=1) [high learner] and P(T|T=0) [low learner] (Pardos & Heffernan, JMLR 2011) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011

Intro to Knowledge Tracing SSI model results 0. 286 RMSE 0. 285 0. 284 Dataset 0. 283 Algebra New RMSE Prev RMSE Improvement 0. 2813 0. 2835 0. 0022 Bridge to Algebra 0. 2824 0. 2860 0. 0036 PPS SSI Average of Improvement is the difference between the 1 st and 3 rd 0. 281 place. It is also the difference between 3 rd and 4 th place. 0. 28 The difference between PPS and SSI are significant in each dataset at the P < 0. 01 level (t-test of squared errors) 0. 279 Algebra Bridge to Algebra (Pardos & Heffernan, JMLR 2011) Zach Pardos Bayesian Knowledge Tracing & Other Models PLSC Summer School 2011