Enhancing Learning with OffTask Social Dialogues Jozef Tvaroek

Enhancing Learning with Off-Task Social Dialogues Jozef Tvarožek and Mária Bieliková Slovak University of Technology in Bratislava EC-TEL 2010, Barcelona September 30, 2010

Our Approach to Socially Intelligent Tutor Enhancing Learning with Off-Task Social Dialogues 2

Learning (1) – Course notes Enhancing Learning with Off-Task Social Dialogues 3

Learning (2) – Problem solving Enhancing Learning with Off-Task Social Dialogues 4

Tasks for assessment and practice Expert’s idea Ability estimate Adaptive selection Task scheme Estimation Generator Answer category Judge Enhancing Learning with Off-Task Social Dialogues Task instance Answer Student 5

Task scheme specification Solution tree Task’s scheme tree A A-C incorrect A-B B C correct B-C Task scheme: • Parameters, constraints, tree of subtasks and answers • Psychometric IRT parameters, usage indicators Enhancing Learning with Off-Task Social Dialogues 6

Task instance Instance generation Parameters’ specification Pruned backtracking Instantiated parameters Scheme tree Instance tree Combine Enhancing Learning with Off-Task Social Dialogues 7

Updating user’s profile • Off-task dialogues – Qs/As scripted to perform actions • • Extracting user’s preferences & behaviors Extracting event attributes Recommending events to attend Negotiating events with others – Relationship maintenance Enhancing Learning with Off-Task Social Dialogues 8

Extracting interests Tutor: Hello Kate, how are you? I'm here to make you feel comfortable, so that you learn much. . . : -) Tutor: tell me more, pls. Student greeting Tutor: ok, write me about yourself, what you like, and all. . . I can then prepare exercises that you will like. . . ; ) Extract features (e. g. to draw, watch TV, friends) < 40 chars ≥ 40 chars Student ack / Turn initiative Enhancing Learning with Off-Task Social Dialogues Tutor: interesting, I for example like to read books, swim, play volleyball and soccer Tutor: now, look around and solve exercises, ok? see you around! 9

Sample conversation Joe: Kate: Joe: Hello Kate, how are you? I’m here to make you feel comfortable, so that you learn much… : -) thnx ok, write me about yourself, what you like, and all… I can then prepare some exercises that you will definitely like : P like to draw sleep watch TV … write a bit more please … like to go out with my dog interesting, I for example like to read books, from sports volleyball, swimming, and also some soccer okay now, look around this environment and solve some of the exercises, ok? see you around! Enhancing Learning with Off-Task Social Dialogues 10

Real-life adaptation of tasks Instance generation, guided by student’s hobbies Parameters’ specification Pruned backtracking Instantiated parameters Scheme tree Semantic similarity with student’s favorite concepts Instance tree Combine Enhancing Learning with Off-Task Social Dialogues 11

Evaluation study • Middle school mathematics • 18 parametric algebra tasks • Tutoring friend – Extract hobbies • Students did participate in a pilot previously – Familiar with the environment Enhancing Learning with Off-Task Social Dialogues 12

Is it better than paper&pencil? • 32 students – Control group = traditional classroom – Experimental group = tutor pre-test post-test gain t-test mean st. dev t stat p-value Control group 0. 697 0. 230 0. 709 0. 211 0. 012 0. 258 -0. 18 0. 428 Exp. group 0. 434 0. 253 0. 538 0. 269 0. 103 0. 172 -2. 40 0. 015 – Learning gain: 1. 2% vs 10. 3% Enhancing Learning with Off-Task Social Dialogues 13

Are they willing to do it? • 16 students • Detect student interests in the initial welcome dialogue: to draw, sleep, watch TV, go out with dog • Mean word count 11. 6 (st. dev 8. 7) • Mean feature count 1. 56 (st. dev 1. 7) • 44% IGNORED the tutor – Others: mfc 2. 78 (st. dev 1. 39) Enhancing Learning with Off-Task Social Dialogues 14

Motivating students? • Those that did engage with the tutor – Less problems attempted, higher success rate. Number of tasks attempted Control Experimental mean st. dev Mean st. dev 7. 71 3. 86 7. 00 1. 80 Number of tasks solved correctly 2. 85 1. 46 4. 00 2. 45 2. 29 1. 25 3. 33 0. 71 2. 29 1. 11 3. 33 1. 12 3. 14 1. 07 4. 33 1. 12 2. 86 1. 07 4. 22 0. 97 Questions (response scale: 1=worst, 5=best) 1. How much did you learn in the tutor? 2. How much did the tutor help you on the post-test? 3. How much would you like to use the tutor again? 4. How did you like the tutor? Enhancing Learning with Off-Task Social Dialogues 15

Motivating students? (contd. ) • Is the tutoring friend any good? – We don’t know. pre-test post-test gain mean st. dev Engaged 0. 429 0. 245 0. 465 0. 283 0. 037 0. 283 Not engaged 0. 439 0. 273 0. 562 0. 284 0. 123 0. 192 – Learning gain: 3. 7% vs. 12. 3% – We can filter students that are engaged, and do well. Enhancing Learning with Off-Task Social Dialogues 16

Summing up • Those who engage in the social off-task dialog with the tutor solve problems better : ) • Tutors that are “friends” with students can produce higher learning gains. • Socially intelligent tutor – tutoring friend: – gets to know you better, – guides you to what you need. Enhancing Learning with Off-Task Social Dialogues 17

Enhancing Learning with Off-Task Social Dialogues Jozef Tvarožek and Mária Bieliková Slovak University of Technology in Bratislava EC-TEL 2010, Barcelona September 30, 2010