Cognitive Tutor Development Kit TDK Tutorial Cognitive Modeling

Cognitive Tutor Development Kit (TDK) Tutorial Cognitive Modeling & Intelligent Tutoring Systems Ken Koedinger Vincent Aleven 1

Overview • Basics of TDK Notation • Matching: $ and = variables • Modeling alternative strategies • “Interface constraints” for model tracing • Control flow and production chains • Writing hints and bug productions • Creating a curriculum 2

Production Rules in the Tutor Development Kit (TDK) • How do you go about writing a production rule? • The TDK Production System – Working Memory: • Make up of “Working Memory Elements” (WMEs) – Production Rules – Interpreter (Match, Conflict Resolution, Fire) 3

Implementing a Production Rule Model in TDK • Simple example: a model for singlecolumn addition without carrying! 4 + 3 7 Done • How would you define: – Working Memory representation for the problem states – Production rules that transform working memory 4

A Notation for Working Memory Elements (“WMEs”) in TDK 4 + 3 WME Name Separator PROBLEM 4+3> ISA FIRST-ADDEND SECOND-ADDEND RESULT DONE WME Type SINGLE-COLUMN-ADDITION-PROBLEM 4 3 nil Slot Values Slots 5

Working Memory Transitions 4 + 3 PROBLEM 4+3> ISA SINGLE-COLUMNADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil Production: ADD 4 + 3 7 PROBLEM 4+3> ISA SINGLE-COLUMNADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE nil Done PROBLEM 4+3> ISA SINGLE-COLUMNADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE T Production: DONE 6

TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem ADD (English version) result NIL IF first-addend =num 1 The goal is to do =problem, a second-addend =num 2 single-column addition problem And no result has been found yet ==> And the first addend is =num 1 And the second added is =num 2 !eval! =sum (+ =num 1 =num 2) THEN Set =sum to the sum of =num 1 and =num 2 =problem> Write =sum as the result =sum ) 7

TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num 1 second-addend =num 2 If-part ==> !eval! =sum (+ =num 1 =num 2) Then-part =problem> result =sum ) 8

TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num 1 second-addend =num 2 If-part WME pattern ==> !eval! =sum (+ =num 1 =num 2) Then-part =problem> result =sum ) Computation in Lisp WME pattern 9

TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num 1 second-addend =num 2 Variables (“bound” to values through matching) WME pattern ==> !eval! =sum (+ =num 1 =num 2) Separator =problem> result =sum ) 10

Matching a Production Rule to Working Memory—Find Values for Each Variable Working Memory Production Rule problem 4+3> add isa single-column=problem> addition-problem isa single-columnfirst-addend 4 addition-problem second-addend 3 result NIL first-addend =num 1 done NIL second-addend =num 2 Match! ==> !eval! =sum (+ =num 1 =num 2) =problem> result =sum ) This WME pattern in the THEN-part means change slot value of existing WME Find value for each variable Variable =problem =num 1 =num 2 =sum Value Problem 4+3 4 3 7 What changes are made to Working Memory? Modify WME problem 4+3> isa single-columnaddition-problem first-addend 4 second-addend 3 result 7 done NIL 11

Summary—Components of a TDK Production Rule Model • Working Memory is a collection of WMEs wme-name> isa wme-type slot 1 value 1 slot 2 value 2 • Production rules specify working memory transitions – WME patterns in the IF-part are matched against Working Memory =wme-variable> isa wme-type slot 1 value-pattern 1 slot 2 value-pattern 2 – WME patterns in the THEN-part specify changes to Working Memory 12

Overview • Basics of TDK Notation • Matching: $ and = variables • Modeling alternative strategies • “Interface constraints” for model tracing • Control flow and production chains • Writing hints and bug productions • Creating a curriculum 13

WME Representation 26 + 51 7 Initial State PROBLEM 26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN 2 COLUMN 1) DONE nil COLUMN 1> ISA COLUMN Value of a slot FIRST-ADDEND 6 SECOND-ADDEND 1 WME can be a RESULT nil COLUMN 2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT nil 26 + 51 77 Done Final State PROBLEM 26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN 2 COLUMN 1) DONE T COLUMN 1> ISA COLUMN in a FIRST-ADDEND 6 list. SECOND-ADDEND 1 RESULT 7 COLUMN 2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT 7 14

Production Rules Set New Goals & Perform Actions 26 + 51 77 Goal: Solve the addition problem FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN DONE Goal: Process column C Action: Mark as done ADD Action: Write the sum 15

Goals in Production Rules FOCUS-ON-FIRST-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column of the problem And there are numbers to add in C And there is no result yet in C THEN Set a subgoal to process column C WRITE-RESULT IF There is a goal to process column C THEN Set Sum to the sum of the addends in column C Write Sum as the result in column C FOCUS-ON-NEXT-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column with numbers to add and no result THEN Set a subgoal to process column C DONE IF The goal is to do a multi-column addition problem And there is no column left with numbers to add and no result THEN Mark the problem as done 16

Figuring Out Which Column FOCUS-ON-FIRST-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column of the problem And there are numbers to add in C And there is no result yet in C THEN Set a subgoal to process column C WRITE-RESULT IF There is a goal to process column C THEN Set Sum to the sum of the addends in column C Write Sum as the result in column C FOCUS-ON-NEXT-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column with numbers to add and no result THEN Set a subgoal to process column C DONE IF The goal is to do a multi-column addition problem And there is no column left with numbers to add and no result THEN Mark the problem as done 17

Example Production Rule (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> FOCUS-ON-NEXT-COLUMN isa column IF - result NIL The goal is to do =problem, a multi-column =next-column> addition problem isa column And =next-column and =previous-column are columns next to each other result NIL And there is a result in =previous-column first-addend =num 1 And there is no result in =next-column second-addend =num 2 And =num 1 is the 1 st addend in =next-column !eval! (or =num 1 =num 2) And =num 2 is the 2 nd addend in =next-column ==> And there is no result yet in =next-column =process-column-goal> And =num 1 or =num 2 is not blank THEN isa process-column-goal Set a subgoal to process =next-column !chain! multi-column-addition (=process-column-goal))

TDK Production Rule Notation If-part Then-part (defproduction focus-on-next-column multi-column-addition (=problem) =problem> WME pattern isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column WME pattern - result NIL =next-column> isa column result NIL WME pattern first-addend =num 1 second-addend =num 2 !eval! (or =num 1 =num 2) ==> =process-column-goal> isa process-column-goal WME pattern column =next-column !chain! multi-column-addition (=process-column-goal))

TDK Production Rule Notation (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> Variables isa column multiple occurrences of result NIL same variable in IF-part first-addend =num 1 must match the same value second-addend =num 2 !eval! (or =num 1 =num 2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal))

TDK Production Rule Notation (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> Variables within a list isa column whose name starts result NIL with $ match any first-addend =num 1 number of items (0 or second-addend =num 2 more). !eval! (or =num 1 =num 2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal))

Matching a Production Rule to Working Memory—Find Values for Each Variable (1) Working Memory Production Rule problem 26+51> focus-on-next-column isa multi-column=problem> addition-problem isa multi-columns (column 2 column 1) addition-problem done nil columns column 1> ($ =next-col =prev-col $) Match! isa column =prev-col> first-addend 6 isa column second-addend 1 - result NIL result 7 =next-col> column 2> isa column result NIL first-addend 2 first-addend =num 1 second-addend 5 second-addend =num 2 result nil !eval! (or =num 1 =num 2) ==> =process-column-goal> Match! isa process-column-goal column =next-col Find value for each variable Variable =problem 1 st $ =next-col =prev-col 2 nd $ =num 1 =num 2 Value problem 26+51 () column 2 column 1 () 6 1 What are made When changes a WME pattern to “isa” Working Memory? with is in the THENpart ofnew a rule, Create WME it means <process-column-goal create a new WME 0>rather isa modify process-column-goal than an existing column 2 one. 22

Pattern Matching with Lists—Patterns with $ variable(s) often create multiple matches Value to be matched: Pattern: (COLUMN 4 COLUMN 3 COLUMN 2 COLUMN 1) ($ =next-column =previous-column $) Match 1 Left $ =next-column =previous-column Right $ () COLUMN 4 COLUMN 3 (COLUMN 2 COLUMN 1) Common pattern to extract the next element of a list. Match 2 Left $ =next-column =previous-column Right $ (COLUMN 4) COLUMN 3 COLUMN 2 (COLUMN 1) Match 3 Left $ =next-column =previous-column Right $ (COLUMN 4 COLUMN 3) COLUMN 2 COLUMN 1 () 23

TDK Production Rule Notation • “==>” separates if-part & then-part • If- & then-parts are made up of WME patterns of form: =wme-variable> isa wme-type slot 1 value-pattern 1 slot 2 value-pattern 2 • An atom starting with either “=” or “$” is a variable – – Ex: “=num” “$columns” “=” vars match atoms “$” variables match 0 or more elements of a list “=” or “$” by itself is just an unnamed (dummy) variable 24

The TDK Production System Interpreter • Match phase: – Attempt to match each WME pattern with 1 or more WMEs in working memory – Variables are “bound” to a value. Subsequent references to those variables must have the same value. – A set of “bindings” for all variables in the if-part is called a “instance” of a production. – If a consistent set of bindings is found, the production “matches”, otherwise it “fails”. – A production can match working memory in more than one way, that is, it can have multiple instances (or rule instantiations). • Conflict resolution phase: – In stand-alone mode: Determined by hand or arbitrarily – In tutor mode: Production priorities & model tracing heuristics • Act phase (fire a production): – Use the variable bindings to modify or create the WME indicated in then-part. – If “isa” slot is present, create a WME, otherwise modify. 25

Overview • Basics of TDK Notation • Matching: $ and = variables • Modeling alternative strategies • “Interface constraints” for model tracing • Control flow and production chains • Writing hints and bug productions • Creating a curriculum 26

Modeling Alternative Strategies • Consider multiple solution paths – Reordering of steps • Write sum first or write carry in next column first – Alternative strategies • Left to right strategy • Use calculator and write result strategy – Shortcuts • Implicit carrying • Which of these will the model cover? • Motivate choices pedagogically! 27

Execution Space Diagram Illustrating Flexibility 1 264 + 716 0 264 + 716 1 264 + 716 80 1 264 + 716 980 264 + 716 0 28

Production Rules Set New Goals & Perform Actions Goal: Solve the addition problem 264 + 716 FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN Goal: Process column C ADD-ADDENDS ADD-CARRY Goal: Write sum in column C MUST-CARRY WRITE-SUM Action: Write the sum Goal: Write carry in next column WRITE-CARRY Action: Write the carry 29

Model Tracing 1 — Using a Cognitive Model to Evaluate Students’ Solution Steps • Cognitive model is an expert system that – is psychologically plausible – accounts for all reasonable solutions – accounts for commonly occurring errors (“bugs”) • To verify solution step by student – Use model to generate correct and buggy next steps – If student step matches • Correct step: Accept – update Working Memory to reflect new problem state • Buggy step: Give error-specific feedback • No step: Reject without feedback 30

Overview • Basics of TDK Notation • Matching: $ and = variables • Modeling alternative strategies • “Interface constraints” for model tracing • Control flow and production chains • Writing hints and bug productions • Creating a curriculum 31

Communicating About Student Solution Steps between Interface and Tutor • Represent each observable action in the user interface as a selection-action-input triple. Selection = (column 2) Action = write-carry Input = <whatever student typed> Selection = (column 1) Action = write-sum Input = <whatever student typed> 32

Execution Space Diagram with Selection-Action-Input Triples Selection: Column 2 Action: Write-Carry Input: 1 1 264 + 716 0 Selection: Column 1 Action: Write-Result Input: 0 1 264 + 716 0 Selection: Column 3 Action: Write-Result Input: 9 1 264 + 716 80 Selection: Column 2 Action: Write-Result Input: 8 1 264 + 716 980 Done. Selection: Action: Done Input: - Selection: Column 2 Action: Write-Carry Input: 1 33

Interface Constraints—Matching Student Steps Against Model Steps • Production rule writer can attach “model -tracing” or “interface constraints” to rules. – Specify tests on the selection-action-input triples. • During model-tracing TDK will not consider productions whose constraints are violated. 34

Example of Interface Constraints in English WRITE-SUM IF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been added THEN Write Sum as the result in column C And remove the goal CONSTRAINTS—consider rule only if: Selection is: Same as value of variable C Action is: Write-Result Input is: Equal to the value of variable Sum Q: Do we really need a constraint on the Selection AND a constraint on the Action AND a constraint on the Input? What if we left out one? A: Yes, each is necessary. Without a constraint on the Selection, the rule will accept the student step if the number typed is correct, regardless of the column in which it was typed. Without a constraint on the Action, the rule will accept the step if the right number was typed in the right column, regardless of whether it was in the carry spot or at the bottom of the column. Etc. 35

Example of Interface Constraints in English WRITE-SUM IF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been added THEN Write Sum as the result in column C And remove the goal CONSTRAINTS—student step matches only if: Selection is: Same as value of variable C Action is: Write-Result Input is: Equal to the value of variable Sum 36

More Detail on Interface Constraints • In Tutor Mode – When evaluating student input • TDK will not consider productions whose constraints are violated. – When responding to a help request • Tries to find the production path with the least severe constraint violation. – from least to most severe: input, action, selection. • In Stand-Alone Mode, interface constraints are ignored • Convenient for development—can run model without having to supply input. 37

TDK Notation for Interface Constraints (defproduction write-sum addition (=problem) =problem> isa addition-problem. . . =subgoal> isa write-sum-goal. . . ==> See Example 4 from the Examples menu. The first item of Selections must be equal to the value of =column> result =sum =problem> subgoals ($sg 1 $sg 2) ; the remaining subgoals : nth-selection 0 =column : action 'write-result : input =sum #'equal-value-p) The Action must be write-result. The value of Input must be equal to the value of =sum. Use function equal-value-p to compare.

Syntax of TDK Notation for Interface Constraints : nth-selection n value [test-fn] : action value [test-fn] : input value [test-fn] 39

Overview • Basics of TDK Notation • Matching: $ and = variables • Modeling alternative strategies • “Interface constraints” for model tracing • Control flow and production chains • Writing hints and bug productions • Creating a curriculum 40

Creating a Control Flow Diagram • Control Flow Diagram elements: – Boxes are goals or actions – Arrows are production rules – A “path” or “chain” through the diagram starts at a goal and ends with an action • Crucial idea: – Each chain should produce exactly one action – When you “cycle” the TDK production system a “chain” of productions fire 41

Production Rules Set New Goals & Perform Actions Goal: Solve the addition problem 264 + 716 FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN Goal: Process column C ADD-ADDENDS ADD-CARRY Goal: Write sum in column C MUST-CARRY WRITE-SUM Action: Write the sum Goal: Write carry in next column WRITE-CARRY Action: Write the carry 42

Chaining of Production Rules —Multiple Productions per Step (defproduction focus-on-next-column (defproduction write-result multi-column-addition (=subgoal) (=problem) =subgoal> =problem> isa process-column-goal isa multi-column-addition-problem No !chain! so column =columns ($ =next-column =previous-column $) this rule ends =column> =previous-column> Name of the chain. isa column production first-addend =num 1 - result NIL rule set second-addend =num 2 =next-column> ==> isa column !eval! =sum (+ (or =num 1 0) (or =num 2 0)) result NIL Parameters =column> first-addend =num 1 result =sum) second-addend =num 2 !eval! (or =num 1 =num 2) ==> !chain! specifies the production set and =process-column-goal> parameters and initiates another attempt to isa process-column-goal column =next-column match productions to this parameter & WM. !chain! multi-column-addition (=process-column-goal)) 43

Important Design Principle • Design your production rule model so that there is exactly one cycle per observable (student) action. 44

Overview • Basics of TDK Notation • Matching: $ and = variables • Modeling alternative strategies • “Interface constraints” for model tracing • Control flow and production chains • Writing hints and bug productions • Creating a curriculum 45

Example Hint Sequences for Addition Tutor 264 + 716 Start with the column on the right. This is the "ones" column. You need to add the two digits in this column. Adding 4 and 6 gives 10. The sum that you have, 10, is greater than 9. You need to 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column. Write 0 at the bottom of the rightmost column. 264 + 716 0 You need to complete the work on the first column. Write the carry from the first to the next column. Write 1 at the top of the second column from the right. 1 264 + 716 Now move on to the second column from the right. This is the "tens" column. You need to add the two digits in this column. Adding 6 and 1 gives 7. There is a carry into this column, so you need to add the value carried in. This gives 7 + 1 equals 8. Write 8 at the bottom of the second column from the right 46

Design of Hint Messages – What makes a good hint sequence? • • • Short Clear (minimum of jargon, short sentences) Tells student what to do But also why/when that action is appropriate Goes from the general to the specific – Writing hints can help in designing the production rule model. – Hint sequences often combine hints from multiple productions in a chain. 47

Example Hint Template (defproduction focus-on-next-column addition (=problem) =problem> isa addition-problem. . . =previous-column> isa column. . . name =name position =pos !eval! (or =num 1 =num 2 =carry) ==>. . . !chain! addition (=problem) Attach list of hint templates to the rule. : messages (help `(Now move on to the #space , =pos column from the right #. ~n ~n This is the #space , =name column #. ))) 48

2 nd Example Hint Template • Example hints 1. Now move on the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! • Each template is a Lisp form that evaluates to a list. : messages (help `(Now move on to the #space , =pos column from the right #. ~n ~n This is the #space , =name column #. ) `(Next big event #: the #space , =pos column from the right #! #! ) ) 49

2 nd Example Hint Template • Example hints 1. Now move on the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! • Each template is a Lisp form that evaluates to a list. : messages (help `(Now move on to the #space , =pos column from the right #. ~n ~n This is the #space , =name column #. ) `(Next big event #: the #space , =pos column from the right #! #! ) ) 50

2 nd Example Hint Template • Example hints 1. Now move on the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! • Each template is a Lisp form that evaluates to a list. : messages (help `(Now move on to the #space , =pos column from the right #. ~n ~n This is the #space , =name column #. ) `(Next big event #: the #space , =pos column from the right #! #! ) ) 51

2 nd Example Hint Template • Example hints from this template 1. Now move on the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! • Each template is a Lisp form that evaluates to a list. : messages (help `(Now move on to the #space , =pos column from the right #. ~n ~n This is the #space , =name column #. ) Backquote. `(Next big event #: the #space , =pos column from the right #! #! ) ) For punctuation To insert value of ~n produces new line. When inserting a string, put a space in front of it. variable: comma followed by variable name. and spaces, use # notation. 52

Sequences Combining Hints from Multiple Productions 264 + 716 0 1 264 + 716 Start with the column on the right. This is the "ones" column. You need to add the two digits in this column. Adding 4 and 6 gives 10. The sum that you have, 10, is greater than 9. You need to move 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column. Write 0 at the bottom of the rightmost column. FOCUS-ON-FIRSTCOLUMN ADD-ADDENDS You need to complete the work on the first column. Write the carry from the first to the next column. Write 1 at the top of the second column from the right. WRITE-CARRY Now move on to the second column from the right. This is the "tens" column. You need to add the two digits in this column. Adding 6 and 1 gives 7. There is a carry into this column, so you need to add the value carried in. This gives 7 + 1 equals 8. Write 8 at the bottom of the second column from the right FOCUS-ON-NEXTCOLUMN ADD-ADDENDS MUST-CARRY WRITE-SUM ADD-CARRY WRITE-SUM 53

How could you model students who don’t carry? • Instead of doing the addition correctly: 1 264 + 716 980 • Can you model a student who writes: 264 + 716 970 • Can you write a “bug rule” that models this error? 54

Production Rules Set New Goals & Perform Actions Goal: Solve the addition problem 264 + 716 FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN Goal: Process column C ADD-ADDENDS ADD-CARRY Goal: Write sum in column C MUST-CARRY WRITE-SUM Action: Write the sum Goal: Write carry in next column WRITE-CARRY Action: Write the carry 55

How to come up with a bug rule? • Intuition: Need to model the “forgetting” of the subgoal to carry into the next column. • Normal processing: 264 + 716 Subgoal 1: write-sum-goal write 0 as result in the first column. Subgoal 2: write-carry-goal write 1 as carry in second column • If you then Forget Subgoal 2, and continue as if nothing has happened • You will pretty soon be in situation: 264 + 716 70 56

Solution 1: Explicit Forgetting Rule ; ; BUG-FORGOT-WRITE-CARRY-GOAL ; ; IF The goal is to do an addition problem ; ; And there is a pending subgoal to write a carry ; ; THEN Delete that pending subgoal (w/o carrying it out). ; ; (And carry on with the addition problem in the usual manner. ) (defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem Use defproduction-bug subgoals ($sg 1 =write-carry-goal $sg 2) instead of defproduction. =write-carry-goal> isa write-carry-goal Use : messages (bug … ) instead ==> of : messages (help … ) =problem> subgoals ($sg 1 $sg 2) ; Remove the write-carry-goal. !chain! addition (=problem) : messages (bug `(First #, you need to write the carry from the previous column #. ) ))

Solution 1: Explicit Forgetting Rule ; ; BUG-FORGOT-WRITE-CARRY-GOAL ; ; IF The goal is to do an addition problem ; ; And there is a pending subgoal to write a carry ; ; THEN Delete that pending subgoal (w/o carrying it out). ; ; (And carry on with the addition problem in the ; ; usual manner. ) Some questions & details • Why does the rule chain? • Why does the rule fire only when the student forgets to add a carry, and not for any wrong answer? • Does this work when action “done” follows after 97+42=39? • Does this rule fire under different circumstances than expected? • YES - after MUST-CARRY in the same cycle. Not intended but not a problem. (defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem subgoals ($sg 1 =write-carry-goal $sg 2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg 1 $sg 2) !chain! addition (=problem) : messages (bug `(First #, you need to write the carry from the previous column #. )) )

Overview • Basics of TDK Notation • Matching: $ and = variables • Modeling alternative strategies • “Interface constraints” for model tracing • Control flow and production chains • Writing hints and bug productions • Creating a curriculum 59

Setting Up a Curriculum • Organize curriculum early on during the development of your tutoring system. • Organize the tutor curriculum – Sequencing of skills – Break down curriculum into lessons and sections – Each section involves a handful of “skills” – Assign problem sets to each section 60

Setting Up Knowledge Tracing and Mastery Learning • Indicate how to group the crucial productions into traced skills. – Each traced skill corresponds to one or more productions. • For each skill, provide values for the four knowledge tracing parameters. • For each problem, indicate which productions are needed to solve it. – Enables tutor to select problems that involve un-mastered skills. 61

Example: Addition Skills • Start — Start in the rightmost column FOCUS-ON-FIRST-COLUMN • Add — Add the addends in a column ADD-ADDENDS • Write — Write the sum in a column WRITE-SUM ADDITION • Done — Determine that all columns are done DONE ADDITION • Next — Focus on the next column FOCUS-ON-NEXT-COLUMN • Regroup — Realize that 10 goes to next column MUST-CARRY • Write Carry — Write the carry WRITE-CARRY • Add Carry — Add the carry to the sum ADD-CARRY ADDITION

Example from Addition Tutor • Structure curriculum, based on skills Section 1 — new skills: Section 2 — new skills: Section 3 — new skills: Section 4 — new skills: one column, no carry Start, Add, Write, Done two columns, no carry Next two columns, possible carry Regroup, Mark Carry, Add Carry three columns, possible carry - 63

Excerpt from Lesson File (Make-Section : Required-Problems '(Problem 11 Problem 12 Problem 15) : Remedial-Problems '(Problem 13 Problem 14 Problem 16 Problem 17 Problem 18 Problem 19) : Skills '(((Focus-On-First-Column Addition) : Description "Start") ((Done Addition) : Description "Done") ((Add-Addends Addition) : Description "Add") ((Write-Sum Addition ) : Description "Write") )) 64

Set Knowledge-Tracing Parameters Parameter Represents probability that (setf *p-initial* 0. 33) the student has learned the skill prior to the instruction the student learns the skill when successfully applying it the student gets the answer right even though the skill is not mastered the student gets the answer wrong even though the skill is mastered Represents mastery threshold (setf *p-learn* (setf *p-guess* (setf *p-slip* 0. 20) 0. 10) (setf *p-known-p* 0. 95) 65

Provide Production Counts For problem 26 + 51 77 (or any other 2 -column problem without carrying) (set-problem-production-counts problem 26+51 '(((FOCUS-ON-FIRST-COLUMN ADDITION). 1) ((FOCUS-ON-NEXT-COLUMN ADDITION). 1) ((ADD-ADDENDS ADDITION). 2) ((WRITE-SUM ADDITION). 2) ((DONE ADDITION). 1) )))) 66

Collecting Logs of Student-Tutor Interactions in TDK • Create “protocol collectors” when starting tutor (make-instance 'global protocol-collector : broadcast-station) (make-instance 'protocol-collector : broadcast-station) ; add to function ; make-addition-tutor • Set directory parameters (setf *global-protocol-directory* "Home: Globals; ") (setf *protocol-directory* "Home: Protocols; ") • Flip the switch (setf *collect-protocols* t) 67

Example Protocol from Addition Tutor *000: 23: 774 GOOD-PATH *000: 17: 53 HELP-PATH *000: 01 START *000: 14: 939 UPDATE-P-KNOW P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), . STUDENT; "Student 3", META; NIL, *000: 23: 778 HISTORY P-2; (ADD-ADDENDS ADDITION), USER-DATA; NIL, PRODUCTION; (FOCUS-ON-FIRST-COLUMN P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-3; (WRITE-SUM ADDITION), PROBLEM; PROBLEM 15, ADDITION), P-2; (ADD-ADDENDS ADDITION), . NPRESENTED; (0 0), SUCCESS? ; NIL, P-3; (WRITE-SUM ADDITION), *000: 17: 68 READY START-TIME; "07/18/01 21: 57", P-KNOW; 0. 3656548145323001, . . FILE-NAME; "Macintosh HD: Desktop. *000: 23: 782 READY *000: 19: 246 SUGGESTED-SELECTION Folder: Vincent: CIRCLE Summer *000: 14: 945 UPDATE-P-KNOW School: TDK: Protocols: LESSON 1: Student 3: 1%PR. ITEMS; (WRITE-RESULT COLUMN 1), META; NIL, OBLEM 15%0. p-0", *000: 25: 84 APPLY-ACTION. PRODUCTION; (ADD-ADDENDS ADDITION), LESSON; LESSON 1, WINDOW; ADDITION-WINDOW, *000: 19: 252 HELP SUCCESS? ; NIL, LESSON-PRINT-NAME; "", CONTEXT; ROOT, SELECTION; (COLUMN 1), P-KNOW; 0. 3656548145323001, SECTION; SECTION 155, SELECTIONS; NIL, NHINT; 1, . SECTION-PRINT-NAME; "", ACTION; DONE, . *000: 14: 952 UPDATE-P-KNOW. INPUT; NIL, #000: 19: 256 HELP-TEXT META; NIL, *000: 24 INITIAL-PROBABILITIES. Write 6 at the bottom of the rightmost column. PRODUCTION; (WRITE-SUM ADDITION), p/p-1; (0. 6762857412475228 (ADD-ADDENDS *000: 25: 94 UPDATE-P-KNOW. SUCCESS? ; NIL, ADDITION)), META; NIL, *000: 19: 259 HELP-PATH P-KNOW; 0. 3656548145323001, p/p-2; (0. 9451767563766541 (DONE ADDITION)), P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), PRODUCTION; (DONE ADDITION), . p/p-3; (0. 6762857412475228 (FOCUS-ON-FIRSTSUCCESS? ; T, P-2; (ADD-ADDENDS ADDITION), COLUMN ADDITION)), *000: 14: 958 NO-CONFLICT-SET P-KNOW; 0. 9898195480465753, P-3; (WRITE-SUM ADDITION), p/p-4; (0. 6762857412475228 (WRITE-SUM. . . ADDITION)), *000: 14: 962 READY *000: 25: 99 GOOD-PATH *000: 19: 270 READY. . *000: 40 FOCUS *000: 17: 41 SUGGESTED-SELECTION *000: 25: 102 HISTORY *000: 21: 94 WINDOW-SELECTED-WMES; (COLUMN 1), ITEMS; (WRITE-RESULT COLUMN 1), P-1; (DONE ADDITION), WINDOW-TITLE; ADDITION-WINDOW, . . *000: 14: 926 APPLY-ACTION *000: 17: 46 HELP *000: 25: 106 READY *000: 23: 765 APPLY-ACTION WINDOW; ADDITION-WINDOW, SELECTION; (COLUMN 1), . WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, NHINT; 0, *000: 25: 109 PROBLEM-COMPLETED CONTEXT; ROOT, SELECTIONS; (COLUMN 1), . . SELECTIONS; (COLUMN 1), ACTION; WRITE-RESULT, #000: 17: 50 HELP-TEXT *000: 25: 113 FINAL-PROBABILITIES ACTION; WRITE-RESULT, INPUT; "5", You need to add the two digits in this column. Adding 3 p/p-1; (0. 3656548145323001 (ADD-ADDENDS INPUT; "6", . and 3 gives 6. ADDITION)), . . p/p-2; (0. 9898195480465753 (DONE ADDITION)), p/p-3; (0. 3656548145323001 (FOCUS-ON-FIRSTCOLUMN ADDITION)), p/p-4; (0. 3656548145323001 (WRITE-SUM ADDITION)), .

Example Uses of Log Data • To improve model and tutor – Is skill decomposition accurate? • If “learning curves” for skills do not follow power law, crucial difficulty factors may have been overlooked. – Tune knowledge tracing parameters • Fit the parameters to the log data, use the new parameter values next time around. – What kind of errors do students make? • Any use for new bug messages? – Do we have enough different problems? • For research – Relation between help use and learning outcomes. – Compare learning rates in comparative studies. 69

END 70