A Student Model for an Intelligent Tutoring System

A Student Model for an Intelligent Tutoring System Helping Novices Learn Object Oriented Design Author: Fang Wei Advisor: Prof. Blank Department of Computer Science Lehigh University May 10, 2007

Presentation Outline n n n Publications Background Research questions Methodology Evaluation Conclusions and future work

Publications n Wei, F. & Blank, G. D. (2007) Atomic Dynamic Bayesian Networks for a Responsive Student Model, Proceedings of the 13 th International Conference on Artificial Intelligence in Education, AIED 2007 Wei, F. & Blank, G. D. (2006) Student Modeling with Atomic Bayesian Networks, Proceedings of the 8 th International Conference on Intelligent Tutoring Systems, ITS 2006, pp. 491502 Wei, F. , Moritz, S. , Parvez, S. , & Blank, G. D. (2005) A Student Model for Object-Oriented Design and Programming. The Journal of Computing Sciences in Colleges (CCSC), Vol. 20, pp. 260 -273. "Best Paper" Award Blank, G. D. , Parvez, S. , Wei, F. , Moritz, S (2005) A web-based n Adaptive Systems for Web-Based Education: Tools and reusability, Amsterdam, The Netherlands, June Moritz, S. , Wei, F. , Parvez, S. , & Blank, G. D. (2005), From n n n ITS for object-oriented design. Poster for 12 th International Conference on Artificial Intelligence in Education, Workshop of objects-first to design-first with multimedia and intelligent tutoring. The Tenth Annual Conference on Innovation and Technology in Computer Science Education (ITi. CSE), Monte da Caparica, Portugal, June.

Presentation Outline n n n Publications Background Research questions Methodology Evaluation Conclusions and future work

Intelligent Tutoring System (ITS) n A computer-based instructional system n n has knowledge bases for instructional content and teaching strategies uses a student’s level of mastery of topics to adapt instruction dynamically A cost-effective means of one-on-one tutoring to provide novices with individualized attention Computer Assisted Instruction (CAI) system does not model what a student is learning and cannot adapt to student n CAI provides same instruction, problems and feedback to every student

Intelligent Tutoring System n Typically contains three main components: An expert evaluator that observes a student’s work and identifies errors in his/her solution n A student model that diagnoses gap in student’s knowledge n A pedagogical advisor that provides feedback to student n

Student Model n n Maintains a model of students’ current knowledge state by representing and updating Provides information for intelligent pedagogical decisions and actions including: curriculum sequencing n interactive problem solving support n pedagogical tutoring customized to each individual student’s learning state n

Motivation n n Novices in high school and college have much difficulty in object oriented design ITSs can aid learners with complex problemsolving n n Design. First-ITS developed to help novices learn object-oriented design (Blank et al. 2005, Moritz & Blank 2005) Research has shown that an ITS that adapts to accurate student knowledge enhances learning (Corbett et al. 2000) Corbett’s ITS focuses on adaptive problem sequencing rather than adaptive feedback n Our ITS focuses on adaptive feedback n

Student Model in Wei & Blank (2006, 2007) compared with other BN Student Models Authors System Context Consider history Murray (1998) Desktop Associate skills Van. Lehn et Solve physics al. (2001, 2005) problems Butz et al. (2004) Reye(1996, 1998, 2004) Theoretical Wei&Blank (2006, 2007) OO Design (UML) analysis √ √ √ Real Time √ √ rules, not concepts C++ programming Millan et CAT for math al. (2002, 2005) Diagnose Pre. Concept requisites No evaluation Postprocess √ √

Layers of Student Knowledge (Self 1994) n n Domain knowledge layer n explain all vocabulary for discussing or solving problems Reasoning knowledge layer n contain reasoning relationships between propositions in domain knowledge Monitoring knowledge layer n specify how to solve a problem using reasoning knowledge and domain knowledge Reflective knowledge layer n specify appropriate strategies students should have in a learning environment

Common Problems with Student Models n n n Do not consider relationship between individual concepts Do not represent layered knowledge (Self 1994) Do not simulate students’ knowledge history Separate the inferred students’ knowledge from closed- and open-ended exercises Do not consider students’ cognitive strategies including general and domain-specific Bayesian student models require exponential updating time and hence cannot provide real-time tutoring adaptive to individual student’s learning state

Presentation Outline n n n Publication Background Research questions Methodology Evaluation Conclusions and future work

Research Questions (1 of 2) Can this student model provide information for pedagogical decisions? n How should this student model represent a student’s current knowledge state and the student’s knowledge structure? n How will the student model track students’ knowledge state over time? Under this research question there are two sub questions: n n n Would tracking a history of students’ knowledge state be useful for pedagogical decisions? Can a history be maintained efficiently enough to be responsive in real-time?

Research Questions (2 of 2) n n How to synthesize information from two different sources, open-ended problem solving (object-oriented class diagram design) and closed-ended exercises (multiple choice quizzes or drag-and-drop exercises)? What cognitive strategies should the student model consider and how to consider them?

Presentation Outline n n n Publication Background Research questions Methodology Evaluation Conclusions and future work

Three Layered Architecture • CM recognizes cognitive strategies that a student is using • HM simulates students’ hierarchical knowledge in a history • PDM simulates current students’ hierarchical knowledge

actor int double_int double int_string actor_object actor_method string numeric datatype object_constructor double_string object datatype object_class datatype_returntype object_method datatype_variable object_attribute class_constructor class_attribute class_method attribute_constructor variable_parameter variable_returntype parameter returntype method_parameter method_constructor method_returntype constructor attribute_method attribute_parameter variable_attribute pass in only A B A is prerequisite of B Curriculum Information Network

Two kinds of concepts n n Unique concept, such as attribute or parameter Relationship concepts, such as attribute_parameter Relationships emerge because of student’s confusions between concepts E. g. , student defines movie. Title as a paramete paramet when he has already defined movie. Title as an attribute

Prerequisite relationships n Prerequisite is relationship between concepts: n n n The concepts a learner needs to understand before understanding a concept E. g. , one needs to understand int and double in order to understand numeric. Datatype Relationship concepts are prerequisites of unique concepts and vice versa n n E. g. , class_constructor -> constructor Understanding constructor doesn’t imply understanding of class, just how to define a constructor for a class

Connecting Knowledge with Performance n Student action unit and knowledge unit make a pair(KU, AU) n n ku A single action or step in a student’s solution E. g. , add an attribute to a class au Action unit (AU): n n n Infer understanding of a concept (KU) from a student solution step (AU) Knowledge unit (KU) – concept a student need to learn n n KU directly causes a student action unit KU is a concept in Curriculum Information Network (CIN)

Atomic Bayesian Network (ABN) d-prereq(ku)1 Noisy-and generalizes logical-and d-prereq(ku)2 ku au …… d-prereq(ku)N Students must understand all direct prerequisites of the concept ku in order to understand ku

How to generate an ABN n n n Student model generates an ABN in response to a student solution step First, define the structure of an ABN, i. e. , the causal relationship between KU and AU, and the direct-prerequisites of KU Second, determine conditional probability tables for this ABN

0 … 0 d-p(ku)N 1 d-p(ku)2 0 d-p(ku)1 … d-p(ku)N 1 d-p(ku)1 ku 0 au 0 ku 1 au 1 Atomic Dynamic Bayesian Network (ADBN) for HM layer

How to generate an ADBN n n Student model generates an ADBN in response to a student solution step First, look for the ABN in response to previous student solution step Second, generate an ABN in response to current student solution step Third, determine conditional probability tables for the ADBN

Concrete Example n Student defined movie. Title as a parameter for method display. Movie. Title after she has already defined movie. Title as an attribute to a class Ticket. Machine n n EE determines that movie. Title should not be a parameter SM determines that the center concept of an ABN is attribute_parameter, and finds all direct prerequisites, attribute and parameter, from CIN

Concrete Example n n n attribute’s prior can be found from the database parameter’s prior is 0. 5, students’ knowledge state is assessed based on the difference between prior and posterior probabilities (Van. Lehn et al. 1998, Millán & Pérez-de-la-Cruz 2002) SM determines: n n student has good understanding of class, attribute, methods, and parameter but low understanding of attribute_parameter the tutoring need is: explanation of attribute_parameter

Concrete Example feedback n “Since you have added movie. Title as an attribute to the class Ticket. Machine, you shouldn’t also make it a parameter to the method display. Movie. Title. To decide whether movie. Title should be an attribute or a parameter, remember: attributes are accessible anywhere within the scope of a class, while parameters are accessible only within the scope of a method”

Presentation Outline n n n Publication Background Research questions Methodology Evaluation Conclusions and future work

Evaluation of ABNs with simulated students n Hypotheses: n n Pre-setting slip and guess values will lead to a reliable student model Varying slip and guess values will affect the accuracy of the student model

n n Pre-setting slip and guess values to same (relatively small e. g. <=0. 1) values produces accuracy of at least 93%, confirming the first hypothesis Changing of presetting slip and guess causes accuracy to change from 79. 1% to 94. 3%, confirming the second hypothesis Correct diagnostic rates are higher when slipp, guessp and slipe, guesse take same value No significant difference when slip and guess take a same small value

Evaluation of ADBNs with simulated students n Hypotheses: n n Pre-setting slip and guess values can lead to a reliable student model Modeling learning history with ADBNs will enhance the accuracy of the student model

n The significant difference between correct diagnostic rates using ABNs versus using ADBNs demonstrates that ADBNs enhance the accuracy of the student model, confirming the second hypothesis

n n Pre-setting slipp/e, guessp/e, to relatively small (e. g. <=0. 1) values produces accuracy of at least 97%, confirming the first hypothesis. The accuracy is not sensitive to change of slip and guess values so along as the values are relatively small (<=0. 1)

Evaluation of student model for CIMEL multimedia with real students n Hypotheses: n n Pre-setting slip, guess will lead to a reliable student model Pre-setting slip and guess to various values will affect the accuracy of the student model The student model will perform in real-time, i. e. , it will support responses as students are working on exercises Results: n The average response time of the student model after a students enters the solution step is 0. 24 seconds

n n Presetting slip and guess with relatively small values (<=0. 1) can produce accuracy of up to 80. 1%. Changing the presetting slip and guess causes the accuracy to change from 71. 7% to 80. 1%

n Has a higher value of r than student model by Corbett et al. (2000)

Evaluation of student model for Design. First-ITS with real students n Hypotheses: n n n Pre-setting slip, guess will lead to a reliable student model The student model will perform in real-time, i. e. , it will support responses as students are working on problem-solving steps Results: n The average response time of the student model after a student enters a solution step is 0. 63 seconds

n n Presetting slip and guess with relatively small values (<=0. 1) can produce accuracy of up to 81. 8% Varying the slip and guess value does not affect the accuracy of the student model, so long as slip and guess values are relatively small (<=0. 1)

n Has a higher value of r than student model by Corbett et al. (2000)

Evaluation of diagnoses integration (multimedia and Design. First-ITS) with real students n Hypothesis: n Integrating diagnoses from closed-ended questions will enhance the accuracy of diagnoses of student model for open-ended questions

n Accuracy increases 7. 7% when adding diagnoses from closed-ended exercises in multimedia, confirming hypothesis

Comparing with non-advancednumerical student models n n n Non-advanced-numerical techniques include match, summation and subtraction Advanced-numerical techniques include Bayesian networks Hypotheses: n n Straightforward algorithm will not lead to a reliable student model ADBNs will perform better than “match” student model

n From same set of evidences, ADBNs perform more than two times better than “match” student models

Presentation Outline n n n n Publications Background Related Research questions Methodology Evaluation Conclusions and future work

Conclusions n n Student models with ADBNs can diagnose student knowledge states accurately in real-time Accuracy of ADBN-based student model is significantly higher than ABN student model Integrating diagnoses from closed- and openended exercises is an effective way to increase accuracy of student models Student models using ADBNs perform much better than the student models that use straightforward algorithm

Future work n n Implement cognitive model to simulate monitoring knowledge and reflective knowledge Consider students learning gain from reviewing feedback n n how do we determine the conditional probability table for the ADBN so as to simulate the real student learning? how do we update the new ADBN? how do we convey empirical studies with simulated students and human subjects? Diagnose students’ learning state in other domains, such as object-oriented programming

An ADBN considers feedback

Contributions (1 of 2) n n n A novel way to represent students’ knowledge structure, where both concepts and relationship between concepts are knowledge units that students need to learn A novel three-layered architecture which can be standardized in modeling various stratums of students’ knowledge ABN – a novel Atomic Bayesian network that provides a refined representation of prerequisite relationships, diagnoses student’s knowledge structure, and guarantees real-time responsiveness

Contributions (2 of 2) n n n ADBN – an innovative dynamic Bayesian network that represents refined representation of prerequisite relationships and diagnoses students’ knowledge structure in real-time considering learning history A unique student model that integrates knowledge from open-ended problem solving (object-oriented class diagram design) and closed -ended exercises A general approach for student models that help students learn complex problem solving in real time

Questions?