Spoken Dialogue for Intelligent Tutoring Systems Opportunities and

Spoken Dialogue for Intelligent Tutoring Systems: Opportunities and Challenges Professor Diane Litman Computer Science Department & Learning Research & Development Center University of Pittsburgh

Outline Motivation and History The ITSPOKE System and Corpora Opportunities and Challenges – Performance Evaluation – Affective Reasoning – Discourse Analysis Summing Up

What is Tutoring? • “A one-on-one dialogue between a teacher and a student for the purpose of helping the student learn something. ” [Evens and Michael 2006] • Human Tutoring Excerpt [Thanks to Natalie Person and Lindsay Sears, Rhodes College]

Intelligent Tutoring Systems Students who receive one-on-one instruction perform as well as the top two percent of students who receive traditional classroom instruction [Bloom 1984] Unfortunately, providing every student with a personal human tutor is infeasible – Develop computer tutors instead

Tutorial Dialogue Systems Why is one-on-one tutoring so effective? “. . . there is something about discourse and natural language (as opposed to sophisticated pedagogical strategies) that explains the effectiveness of unaccomplished human [tutors]. ” [Graesser, Person et al. 2001] Working hypothesis regarding learning gains – Human Dialogue > Computer Dialogue > Text

Spoken Tutorial Dialogue Systems Most human tutoring involves face-to-face spoken interaction, while most computer dialogue tutors are text-based Can the effectiveness of dialogue tutorial systems be further increased by using spoken interactions?

Potential Benefits of Speech: I Self-explanation correlates with learning [Chi et al. 1994] and occurs more in speech [Hausmann and Chi 2002] – Tutor: The right side pumps blood to the lungs, and the left side pumps blood to the other parts of the body. Could you explain how that works? – Student 1 (self-explains): So the septum is a divider so that the blood doesn't get mixed up. So the right side is to the lungs, and the left side is to the body. So the septum is like a wall that divides the heart into two parts. . . it kind of like separates it so that the blood doesn't get mixed up. . . – Student 2 (doesn’t self-explain): right side pumps blood to lungs

Potential Benefits of Speech: I Self-explanation correlates with learning [Chi et al. 1994] and occurs more in speech [Hausmann and Chi 2002] – Tutor: The right side pumps blood to the lungs, and the left side pumps blood to the other parts of the body. Could you explain how that works? – Student 1 (self-explains): So the septum is a divider so that the blood doesn't get mixed up. So the right side is to the lungs, and the left side is to the body. So the septum is like a wall that divides the heart into two parts. . . it kind of like separates it so that the blood doesn't get mixed up. . . – Student 2 (doesn’t self-explain): right side pumps blood to lungs

Potential Benefits of Speech: I Self-explanation correlates with learning [Chi et al. 1994] and occurs more in speech [Hausmann and Chi 2002] – Tutor: The right side pumps blood to the lungs, and the left side pumps blood to the other parts of the body. Could you explain how that works? – Student 1 (self-explains): So the septum is a divider so that the blood doesn't get mixed up. So the right side is to the lungs, and the left side is to the body. So the septum is like a wall that divides the heart into two parts. . . it kind of like separates it so that the blood doesn't get mixed up. . . – Student 2 (doesn’t self-explain): right side pumps blood to lungs

Potential Benefits of Speech: II Speech contains prosodic information, providing new sources of information about the student for dialogue adaptation [Fox 1993; Litman and Forbes-Riley 2003; Pon-Barry et al. 2005] A correct but uncertain student turn – ITSPOKE: How does his velocity compare to that of his keys? – STUDENT: his velocity is constant

Potential Benefits of Speech: III Spoken computational environments may foster social relationships that may enhance learning – Auto. Tutor [Graesser et al. 2003]

Potential Benefits of Speech: IV • Some applications inherently involve spoken language – Spoken Conversational Interface for Language Learning [MIT(Seneff, Glass, Wang), Cambridge (Young, He, Ye)] – Reading Tutors [Mostow, Cole] • Others require hands-free interaction – Circuit Fix-It Shop [Smith 1992]

Why Should NLP Researchers Care? Many reasons why tutoring researchers are interested in spoken dialogue Why should researchers in computational linguistics become interested in tutoring? – Tutoring applications differ in many ways from typical spoken dialogue applications – Opportunities and Challenges!

Outline Motivation and History The ITSPOKE System and Corpora Opportunities and Challenges – Performance Evaluation – Affective Reasoning – Discourse Analysis Summing Up

• Back-end is Why 2 -Atlas system [Van. Lehn, Jordan, Rose et al. 2002] • Sphinx 2 speech recognition and Cepstral text-to-speech

• Back-end is Why 2 -Atlas system [Van. Lehn, Jordan, Rose et al. 2002] • Sphinx 2 speech recognition and Cepstral text-to-speech

• Back-end is Why 2 -Atlas system [Van. Lehn, Jordan, Rose et al. 2002] • Sphinx 2 speech recognition and Cepstral text-to-speech

Two Types of Tutoring Corpora Human Tutoring – 14 students / 128 dialogues (physics problems) – 5948 student turns, 5505 tutor turns Computer Tutoring – ITSPOKE v 1 » 20 students / 100 dialogues » 2445 student turns, 2967 tutor turns – ITSPOKE v 2 » 57 students / 285 dialogues » both synthesized and pre-recorded tutor voices

ITSPOKE Experimental Procedure College students without physics – Read a small background document – Took a multiple-choice Pretest – Worked 5 problems (dialogues) with ITSPOKE – Took an isomorphic Posttest Goal was to optimize Learning Gain – e. g. , Posttest – Pretest

Outline Motivation and History The ITSPOKE System and Corpora Opportunities and Challenges – Performance Evaluation – Affective Reasoning – Discourse Analysis Summing Up

Predictive Performance Modeling Opportunity – Spoken dialogue system evaluation methodologies can improve our understanding of how dialogue facilitates student learning [Forbes-Riley and Litman 2006] Challenges – How to measure system performance? – What are predictive interaction parameters?

Predictive Performance Modeling Understand why a spoken dialogue system fails or succeeds PARADISE [Walker et al. 1997] – Measure parameters (interaction costs and benefits) and performance in a system corpus – Train model via multiple linear regression over parameters, predicting performance n System Performance = ∑ wi * pi – Test model on new corpus i=1 – Predict performance during future system design

Challenges System Performance – Prior evaluations used User Satisfaction – Is Student Learning more relevant for the tutoring domain? Interaction Parameters – Prior applications used Generic parameters – Are Task-Specific and Affective parameters also useful?

Findings Using PARADISE to predict Learning – Posttest =. 86 * Time +. 65 * Pretest -. 54 * #Neutrals Traditional predictive parameters – e. g. , Elapsed Time, Dialogue and Turn Length New parameters – e. g. , Affect, Correctness Predictive power increases with the linguistic sophistication of the parameters – e. g. , Semantic concepts rather than words

Contrasts with Non-Tutorial Dialogue User Satisfaction models are less useful – Tutoring systems are not designed to maximize User Satisfaction Interaction parameters for learning – Posttest =. 86 * Time +. 65 * Pretest -. 54 * #Neutrals » longer dialogues are better » speech recognition problems don’t seem to matter » lack of some types of affect is bad

Contrasts with Non-Tutorial Dialogue User Satisfaction models are less useful – Tutoring systems are not designed to maximize User Satisfaction Interaction parameters for learning – Posttest =. 86 * Time +. 65 * Pretest -. 54 * #Neutrals » longer dialogues are better » speech recognition problems don’t seem to matter » lack of some types of affect is bad

Contrasts with Non-Tutorial Dialogue User Satisfaction models are less useful – Tutoring systems are not designed to maximize User Satisfaction Interaction parameters for learning – Posttest =. 86 * Time +. 65 * Pretest -. 54 * #Neutrals » longer dialogues are better » speech recognition problems don’t seem to matter » lack of some types of affect is bad

Outline Motivation and History The ITSPOKE System and Corpora Opportunities and Challenges – Performance Evaluation – Affective Reasoning – Discourse Analysis Summing Up

Detecting and Responding to Student Affective States Opportunity – Affective spoken dialogue system technology can improve student learning and other measures of performance [Aist et al. 2002; Pon-Barry et al. 2006] Challenges – What to detect? – How to respond?

Monitoring Student State (motivation) Tutor 26: Let me ask my question in a more general way. What happens to the velocity of a body when there is no force acting on it? Student 27: dammit (ASR: it is) Tutor 28 : Could you please repeat that? Student 29 : same (ASR: i same) Tutor 30 : Right. And what happens to the HORIZONTAL velocity of a body when there is no HORIZONTAL force acting on it? Student 31: zero (ASR: the zero) Tutor 32 : Well. . . OK, let's consider how forces in the vertical direction affect the motion of a body in a horizontal direction in a different scenario <…omitted…> Student 33: oh gravity you already said this (ASR: beats gravity you're exert less) Tutor 34: Fine. Are there any other forces acting on the apple as it falls? Student 35: no why are you doing this again (ASR: no y and to it yes) Tutor 36: Correct. The only force acting on the apple as it falls to the earth is the earth's gravitational force. In which direction does gravitational force act? Student 37: downward you computer (ASR: downward you computer)

Affective Spoken Dialogue Systems: Standard Methodology Manual Annotation of Affect and Attitudes – Naturally-occurring spoken dialogue data [Ang et al. 2002; Lee et al. 2002; Batliner et al. 2003; Devillers et al. 2003; Shafran et al. 2003; Liscombe et al. 2005] Prediction via Machine Learning – Automatically extract features from user turns – Use different feature sets (e. g. prosodic, lexical) to predict affect – Significant reduction of baseline error

Challenge 1: What “emotions” to detect? Communicator and Customer Care Systems – Negative: Angry, Annoyed, Frustrated, Tired – Positive/Neutral: Amused, Cheerful, Delighted, Happy, Serious [Ang et al. 2002; Shafran et al. 2003; Lee and Narayanan 2005; Liscombe et al. 2005]

Challenge 1: What “emotions” to detect? Communicator and Customer Care Systems – Negative: Angry, Annoyed, Frustrated, Tired – Positive/Neutral: Amused, Cheerful, Delighted, Happy, Serious [Ang et al. 2002; Shafran et al. 2003; Lee and Narayanan 2005; Liscombe et al. 2005] Tutorial Dialogue Systems – Negative: Angry, Annoyed, Frustrated, Bored, Confused, Uncertain, Contempt, Disgusted, Sad – Positive/Neutral: Certain, Curious, Enthusiastic, Eureka [Litman and Forbes-Riley 2006, D’Mello et al. 2006]

Example Student States in ITSPOKE: What else do you need to know to find the box‘s acceleration? Student: the direction [UNCERTAIN] ITSPOKE : If you see a body accelerate, what caused that acceleration? Student: force [CERTAIN] ITSPOKE : Good job. Say there is only one force acting on the box. How is this force, the box's mass, and its acceleration related? Student: velocity [UNCERTAIN] ITSPOKE : Could you please repeat that? Student: velocity [ANNOYED]

Challenge 2: How to respond? • In tutoring, not all negatively-valenced emotions are bad! – While frustration/anger/annoyance is often frustrating… – Frustration can also be an opportunity to learn

Example from ITSPOKE: What is the net force acting on the truck equal to? Student: I don’t know [Uncertain, Frustrated] ITSPOKE: Since net force is defined as the vector sum of all the forces exerted on an object, let's first find the forces acting on the truck. Try to name them. Student: The impact force, the normal force, and gravity [Certain, Neutral] ITSPOKE: Yep.

Uncertainty is also a Learning Opportunity Uncertainty represents one type of learning impasse [Van. Lehn et al. 2003]: An impasse motivates a student to take an active role in constructing a better understanding of the principle. Uncertainty is also associated with cognitive disequilibrium [Craig et al. 2004]: A state of failed expectations causing deliberation aimed at restoring equilibrium – Uncertainty positively correlates with learning

Do Human Tutors Respond to Student Uncertainty? A data-driven method for designing dialogue systems adaptive to student state [Forbes-Riley and Litman 2005] – extraction of “dialogue bigrams” from annotated human tutoring corpora – χ2 analysis to identify dependent bigrams – generalizable to any domain with corpora labeled for user state and system response

Example Human Tutoring Excerpt S: T: So the- when you throw it up the acceleration will stay the same? [Uncertain] Acceleration uh will always be the same because there isthat is being caused by force of gravity which is not changing. [Restatement, Expansion] mm-k. [Neutral] Acceleration is– it is in- what is the direction uh of this acceleration- acceleration due to gravity? [Short Answer Question] S: T: It’s- the direction- it’s downward. [Certain] Yes, it’s vertically down. [Positive Feedback, Restatement]

Bigram Dependency Analysis - “Student Certainness – Tutor Positive Feedback” Bigrams χ2 = 225. 92 (critical χ2 value at p =. 001 is 16. 27) Tutor Includes. Pos Tutor Omits. Pos neutral 252 2517 neutral 439. 46 2329. 54 certain 273 832 certain 175. 21 928. 79 uncertain 185 631 uncertain 129. 51 686. 49 mixed 71 161 mixed 36. 82 195. 18 OBSERVED EXPECTED Tutor Include. Pos Omits. Pos

Bigram Dependency Analysis (cont. ) - Less Tutor Positive Feedback after Student Neutral turns OBSERVED neutral Includes Omits Pos 252 2517 EXPECTED neutral Includes Omits Pos 439. 46 2329. 54

Bigram Dependency Analysis (cont. ) - Less Tutor Positive Feedback after Student Neutral turns - More Tutor Positive Feedback after “Emotional” turns Includes Omits Pos neutral 252 2517 certain 273 uncertain mixed OBSERVED Includes Omits Pos neutral 439. 46 2329. 54 832 certain 175. 21 928. 79 185 631 uncertain 129. 51 686. 49 71 161 mixed 36. 82 195. 18 EXPECTED

Findings Statistically significant dependencies exist between students’ state of certainty and the responses of an expert human tutor – After uncertain, tutor Bottoms Out & avoids expansions – After certain, tutor Restates – After mixed, tutor Hints – After any emotion, tutor increases Feedback Dependencies suggest adaptive strategies for implementation in computer tutoring systems – Experiments with adaptive ITSPOKE

Adapting to Student Disengagement is also of Interest • User sings answer indicating lack of interest in its purpose ITSPOKE: What vertical force is always exerted on an object near the surface of the earth? USER: Gravity (disengaged, certain)

Affect-Adaptive ITSPOKE: Results Responding to student uncertainty (over and above correctness) improves ITSPOKE’s performance [Forbes-Riley and Litman 2011 a, b] – Student learning – Dialogue efficiency Responding to student disengagement (over and above uncertainty) even further improves performance [Forbes-Riley and Litman 2012]

Outline Motivation and History The ITSPOKE System and Corpora Opportunities and Challenges – Performance Evaluation – Affective Reasoning – Discourse Analysis Summing Up

Discourse Structure Opportunity – Dialogues with tutoring systems have more complex hierarchical discourse structures compared to many other types of dialogues Challenges – How can discourse structure be exploited in the context of spoken dialogue systems?

Exploiting Discourse Structure (Motivation) Average ITSPOKE dialogue is 20 minutes Student turns are hierarchically structured – Level 1 : 1350 – Level 2 : 643 – Level 3 : 248 – Levels 4 -6 : 113 (57. 3%) (27. 3%) (10. 5%) (4. 8%)

Discourse structure Annotation and Transitions Based on the Grosz & Sidner theory of discourse structure – Discourse segment purpose – Hierarchy of discourse segments Tutoring information encoded in a hierarchical structure – Human tutor manually authored dialogue paths for ITSPOKE – Automatic traversal of logs places utterances into the structure Q 1 Q 2. 1 Q 3 Q 2. 2

ITSPOKE behavior & Discourse structure annotation Q 1 Q 2. 1 Q 3 Q 2. 2

Discourse structure transitions Q 1 Q 2. 1 Q 3 Q 2. 2

Findings Student correctness is predictive of student learning, but only after particular discourse transitions [Rotaru and Litman 2006] – e. g. , After Pops (Pop. Up, Pop. Up. Advance) » incorrect turns negatively predict learning » correct turns positively predict learning – Currently testing with experimental manipulation Student certainness is more predictive only after particular transitions

Findings (cont. ) While single discourse transitions are not predictive of learning, patterns in the discourse structure are – e. g. , Advance-Advance and Push-Push both positively correlate with learning Statistically significant dependencies exist between discourse transitions and speech recognition – e. g. , after both Pushes and Pops, more misrecognitions Graphical display of discourse structure increases user satisfaction – e. g. , easier for students to concentrate and to learn

Outline Motivation and History The ITSPOKE System and Corpora Opportunities and Challenges – Performance Evaluation – Affective Reasoning – Discourse Analysis Summing Up

Summing Up: I Spoken Dialogue Systems are of great interest to researchers in Intelligent Tutoring – One-on-one tutoring is a powerful technique for helping students learn – Natural language dialogue contributes in a powerful way to the efficacy of one-on-one-tutoring – Using presently available NLP technology, computer tutors can be built and can serve as a valuable aid to student learning

Summing Up: II Intelligent Tutoring in turn provides many opportunities and challenges for researchers in Spoken Dialogue Systems – Performance Evaluation – Affective Reasoning – Discourse Analysis

Summing Up: II Intelligent Tutoring in turn provides many opportunities and challenges for researchers in Spoken Dialogue Systems – Performance Evaluation – Affective Reasoning – Discourse Analysis – and many more! » Cohesion/Coherence, Priming and Alignment, Dialogue Acts, Reinforcement Learning, User Simulation, Prosody and Dialogue

Next Opportunity/Challenge?

Acknowledgements ITSPOKE group NLP@Pitt (R. Hwa, J. Wiebe et al. ) Why 2 -Atlas and Rimac Tutoring groups (K. Van. Lehn, P. Jordan, C. Rose, S. Katz et al. ) Engineering Design group (C. Schunn et al. )

Thank You! Questions? Further Information – http: //www. cs. pitt. edu/~litman/itspoke. html Annotated ITSPOKE Corpus – https: //learnlab. web. cmu. edu/datashop/index. jsp

The End

Detecting Neg/Pos/Neu in ITSPOKE - As with other applications, highest predictive accuracies are obtained by combining multiple feature types [Litman and Forbes-Riley 2006]

Detecting Neg/Pos/Neu in ITSPOKE - However, relative feature utility differs in tutoring (e. g. , for speech features: temporal > energy > pitch)

Detecting and Responding to Student Affective States Opportunity – Affective spoken dialogue system technology can improve student learning and other measures of performance [Aist et al. 2002; Pon-Barry et al. 2006] Challenges – What to detect? – How to respond? – Pedagogical versus spoken dialogue performance?

Challenge 3: Pedagogical versus spoken dialogue performance? Negative user emotions (e. g. frustration) are often associated with speech recognition problems [Boozer et al. 2003; Goldberg et al. 2003] – Is this also true in tutoring? Speech recognition problems negatively correlate with user satisfaction [Walker et al. 2002, Pon-Barry et al. 2006] – Is this also true for learning?

Findings Statistically significant dependencies exist between student state and speech recognition problems [Rotaru and Litman 2006] – Frustrated/Angry turns are rejected more than expected – Uncertain turns have more problems than expected (certain turns have less) – Incorrect turns have more problems than expected (correct turns have less) Learning opportunities (e. g. uncertain and incorrect student states) have more speech recognition problems – However, speech recognition problems have not negatively correlated with learning [Litman and Forbes-Riley 2005, Pon-Barry et al. 2005]

In Closing Synergy between Intelligent Tutoring and Spoken Dialogue Systems can provide – Better scientific understanding of how dialogue facilitates learning – Long-term benefit for scaling spoken dialogue systems to new and complex domains