Bayes Net Toolbox for Student Modeling BNTSM Kaimin
Bayes Net Toolbox for Student Modeling (BNT-SM) Kai-min Chang & Yanbo Xu EDM 2015 SMART Tutorial June 26, 2015 This work was supported by the National Science Foundation under Cyberlearning Grant IIS 1124240. The opinions expressed are those of the authors and do not necessarily represent the views of the Institute, or the National Science Foundation.
Download • BNT-SM – http: //www. cs. cmu. edu/~listen/BNT-SM • Matlab – Check your university’s license • Octave – http: //www. gnu. org/software/octave 2
Bayes Net Toolkit for Student Modeling (BNT-SM) Bayes. Net. xml Observations of student performance BNT-SM Estimates of model parameters BNT Estimates of student knowledge 3
Model Knowledge Tracing with Some Bayes Nets Packages • Popular Bayes Nets packages (at the time…) – BNT, BUGS, GMTK – Complicated coding effort to use the packages • Create toolkit to hide the complexity from ITS researchers. – Extend Bayes Nets Toolkit (BNT; Murphy, 1998) 4
Revision History • First developed 9 years ago (Chang et al. , 2006) • Incorporated LR-DBN (Xu & Mostow, 2011) • Updated for the tutorial (SMART, 2015) – To run on Octave (used a workaround since xmlread. m is not implemented in Octave). – Attempted to use the most updated BNT (unsuccessful). 5
Knowledge Tracing (KT) • Corbett & Anderson (1995) • Infers a student’s proficiency on a skill from observation of applying the skill. • Maintain 4 skill-specific parameters: – Learning parameters: Already know & learn – Performance parameters: guess & slip Knowledge of the skill “cat” Correct 0. 40 0. 32 0. 21 0. 29 0. 45 0. 38 6
Bayesian Knowledge Tracing (BKT) Already know Student Knowledge (K 0) Affect current attempt Affect future attempt learn, forget Student Knowledge (Kn) guess, slip Student Performance (C 0) Student Performance (Cn) 7
Bayes Net Toolkit for Student Modeling (BNT-SM) Step 1 Bayes. Net. xml Step 4 Step 3 BNT-SM Step 2 Observations of student performance Estimates of model parameters Step 5 BNT 8 Estimates of student knowledge
1 a. Specify Network Structure Student Knowledge (K 0) Kn Student Performance (C 0) Cn <nodes> <node> <name>Student. Knowledge</name> <values>2 </values> <type> discrete </type> <observed> latent</observed> <within> Student. Performance </within> <between>Student. Knowledge </between> </node> …… 9
1 b. Initialize Model Parameters Already know Student Learn Knowledge Kn (K 0) Guess, Slip Student Performance (C 0) XML Cn 10
2. Provide Observations of Student Performance Ordered attempts within a skill Comprehensive observational data • • Skill User my Kevin 2002 -08 -14 15: 30: 21 ? cat Kevin 2002 -08 -14 15: 30: 22 ? … … Performance Time Student Knowledge Performance … 11
3. Run BNT-SM Bayes. Net. xml Observations of student performance BNT-SM Estimates of model parameters BNT Estimates of student knowledge 12
4. Examine Model Parameters Parameter values can be used to infer which skill is more difficult Skill # Users # Cases LL L 0 Guess Slip Learn Forget My 14 23 -5. 15 0. 46 0. 73 0. 04 0. 20 0. 00 Cat 46 219 -94. 07 0. 44 0. 70 0. 11 0. 24 0. 00 Average log likelihood can be used to compare across different models 13
5. Examine Model Inferences User Skill Knowledge Correct Alice My 0. 463045 NULL Bob My 0. 463045 NULL Cathy … My … 0. 463045 … NULL … John Cat 0. 440226 NULL Kevin … Cat … 0. 620417 … NULL … Estimates the progression of student knowledge that can be used by the tutor 14
Matlab / Octave >> cd BNT-SM/src >> setup >> cd. . /model/kt Sample Bayes net models >> [property evidence hash_bnet] = Run. Bnet('property. xml'); An XML file that specifies the Bayes net we are constructing. 15
Examine Some Files >> vim property. xml (Matlab) >> vim load_property. m (Octave) Input >> vim evidence. train. xls >> vim evidence. test. xls >> vim param_table. xls >> vim inference_result_header. xls >> vim inference_result. xls 16 Output
Model 1: KT • Corbett & Anderson (1995) l 0 K(1) … l f l K(t) f K(t+1) g s g s C(1) C(t+1) 17 …
Model 1: KT • Corbett & Anderson (1995) l 0 ? … l f ? g s g s 18 …
Model 2: LR-DBN • An Extension of BNT-SM to trace multiple subskills (Xu & Mostow, 2012) S 1(t+1) S 1(t) S 2(t+1) S 2(t) K(1) … le fe K(t) … … l 0 Sm(t) le fe K(t+1) g s g s C(1) C(t+1) 19 Sm(t+1) …
Model 3: EEG-KT • Insert a binary EEG measure into KT (Yuan et al. , 2014) l 0 … K(1) le fe le K(t) E(1) ge se fe E(t+1) E(t) ge se C(t) C(t+1) ge se C(1) 20 … K(t+1)
Model 3: EEG-KT • Insert a binary EEG measure into KT (Yuan et al. , 2014) l 0 ? … fe ge se Idea Model le Evaluate ? le fe ? ge se Discuss 21 …
References • Chang, K. M. , Beck, J. E. , Mostow, J. , & Corbett, A. (2006). A Bayes net toolkit for student modeling in intelligent tutoring systems. Proceedings of the 8 th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, 104 -113. • Xu, Y. , & Mostow, J. (2011). Using Logistic Regression to Trace Multiple Subskills in a Dynamic Bayes Net. Proceedings of the 4 th International Conference on Educational Data Mining Eindhoven, Netherlands, 241 -245. • Yuan, Y. , Chang, K. M. , Xu, Y. , & Mostow, J. (2014). A Public Toolkit and ITS Dataset for EEG. Proceedings of the 12 th International Conference on Intelligent Tutoring Systems Workshop on Utilizing EEG Input in Intelligent Tutoring Systems, Honolulu, Hawaii. 22
Questions? • Kai-min Kevin Chang – Language Technologies Institute, Carnegie Mellon University – E-mail: kkchang@cs. cmu. edu • Yanbo Xu – Robotics Institute, Carnegie Mellon University – E-mail: yanbox@cs. cmu. edu 23
- Slides: 23