ConceptLevel Topic Analysis of Tutoring Dialogs Amruta Purandare

Concept-Level Topic Analysis of Tutoring Dialogs Amruta Purandare, Intelligent Systems Program Dr. Diane Litman, Department of Computer Science 3. Methodology 1. Background: ITSPOKE 6. Good! Revise your essay now ^^^^^^^^^^^^ 2. Done! Automatic Topic Extraction 4. What are the forces acting on the pumpkin? • Topics = Domain-Specific Concepts • Online Physics Encyclopedia ‘Eric Weisstein's World of Physics’ 3. You need more details ^^^^^^ • Intelligent Tutoring SPOKEn dialog-based system • Interactive Dialog Sessions teaching Qualitative Physics • Backend: Why-2 text-based physics tutoring [Van. Lehn et. al. ] • Tutor’s Role: • Present a problem • Clarify missing points, Correct misconceptions • Ask Q’s to check understanding • Request to revise answer • Student’s Role: -Write Essays -Answer Q’s • Present analysis limited to single-word concepts 2. Motivation: What Affects Learning? Previous Work • Shallow Metrics (Automatic) [Litman et. al. ITS 04] • Dialog Length (#turns, #words, words/turn) • Distinguish Student and Tutor Contributions • Deep Metrics (Manually Labeled) • Student and Tutor Actions [Forbes-Riley et. al. AIED 05] Misses the central theme of discussion: “Topics” or “Content” Challenges • Granularity • Simple, Complex, Abstract Topics? • 1 topic per turn/sentence? • No Manual Annotations • #physics-concepts, #critical-concepts discussed per student • #dialog turns that discuss n concepts (where n = 0, 1, 2, 3+, 1+) • Average #physics-concepts per turn Word Level Correlations S+T+E S+T S+E S T E #WORDS R=0. 44 R=0. 297 R=0. 516 R=0. 393 R=0. 287 R=0. 504 • Ambiguity: “current problem” Vs “electric current” #PHY CONCEPTS R=0. 488 R=0. 389 R=0. 557 R=0. 576 R=0. 355 R=0. 519 • 36 physics concepts observed in our data CONCEPTS/WORDS R=0. 473 R=0. 505 R=0. 429 R=0. 396 R=0. 485 R=0. 329 • 10 concepts appear in ideal answers are marked as ‘Critical’ #CRITICAL R=0. 52 R=0. 393 R=0. 55 R=0. 511 R=0. 364 R=0. 515 • Analyze number of times each concept is repeated per student Example Tutor: You need to show even more of your reasoning in your essay. What is the name of the principle of physics that you could apply to determine the difference in acceleration between the car and truck given that they both experience the same force? Student: I don’t know Our work • Show: Topics correlate to Learning • Automatically identify Topic Structures in Dialogs Metrics • Contains total 3040 terms including multi-word concepts such as “Frictional Force”, “Newton’s second law” etc. 5. Gravity? 1. Solve this problem 4. Preliminary Results Tutor: Newton's Second Law. We know that Newton's Second Law expresses a relationship between force, mass, and acceleration. What is the equation for that relationship? Student: Force equals mass times acceleration Turn Level Correlations #CONCEPTS/TURN S+T+E S+T S+E S T E R=0. 033 ZERO R=0. 028 R=0. 027 R=0. 055 R=-0. 05 ONE R=0. 439 R=0. 466 R=0. 508 R=0. 584 R=0. 243 R=-0. 217 TWO R=0. 462 R=0. 489 R=0. 217 R=0. 303 R=0. 465 R=-0. 04 THREE and MORE R=0. 326 R=0. 252 R=0. 358 R=0. 445 R=0. 222 R=0. 314 AT LEAST ONE R=0. 443 R=0. 445 R=0. 582 R=0. 607 R=0. 347 R=0. 238 AVERAGE R=0. 323 R=0. 367 R=0. 261 R=0. 505 R=0. 287 R=0. 328 Bold indicates correlations significant at 0. 05 level Bold and Italics indicates correlations significant at 0. 01 level S = Student (spoken), T = Tutor (spoken), E = Essay (written) Tutor: Fine. Applying this principle, if you apply equal force to two objects of differing masses, you can use the relative accelerations of the two objects to determine the relationship between the respective masses of the two objects. In this case, which vehicle will have a smaller acceleration? • Physics concepts show better correlations than all words Student: Truck • #Dialog turns with 0 and 3+ concepts show no impact Tutor: Uh huh. So then you know that during the impact, the magnitude of the acceleration of the truck is less than the magnitude of the acceleration of the car for every time instance. Try to add something about this to your essay. Evaluation • Students take tests before (pre) and after (post) tutoring • We study partial correlations (R) between post-test scores and concept-metrics, controlled for pre-test scores Conclusions • Student contributions matter! 5. Current Directions • Include Multi-Word Concepts e. g. “Newton’s Second Law” • Do non-physics words have any role in learning? • Analyze Discourse-Level Topic Structures and Segments