Conversational Agents and Learning Outcomes An Experimental Investigation

Conversational Agents and Learning Outcomes: An Experimental Investigation Bob Heller Mike Proctor Athabasca University Research supported by Mission Critical Research Grant - Athabasca University EDMEDIA, Vancouver, 2007

Conversational agents as pedagogical agents • Auto. Tutor - conversational intelligent tutor system (see Graesser, Wiemer-Hastings, Kreuz, & Tutoring Research Group 1999) • Embodied Conversational Agents - primacy of conversation (Cassells et al) • Pedagogical Agent research juncture of HCI and educational psychology (Hadwin, Winne, & Nesbit, 2005)

The Persona Effect Pedagogical agents have the potential to increase the bandwidth of communication between students and computers and the computer’s ability to engage and motivate students (Johnson, Rickel, and Lester, 2000) The Media Equation Reeves & Nass

Evidence for Persona Effect

Objectives of the Present Study Learning Outcome Historical Figure replication

Method Participants Materials Procedure

Logical Interface

Chat Interface

Text Interface Evaluation Chat Interface n=28 n=31 M How enjoyable was this activity? How engaging was this activity How easy was this activity? Would you recommend this activity to others? Overall, how would you rate this activity? How useful is this interface for learning information about Jean Piaget? How useful is this interface for remembering information about Jean Piaget? M SD t 3. 43 3. 11 3. 68 3. 43 2. 87 3. 19 2. 97 2. 87 2. 32* 0. 79 0. 32 2. 54* 2. 23* 4. 22 3. 42 2. 80** 3. 53 3. 00 2. 49* Table 1: Interface evaluation ratings based on a 5 -point scale where higher numbers reflect positive experiences. Note: * -p <. 05 ; ** - p <. 01

Text Chat p 46. 9% 83. 8% 67. 4% Non-exposed Content 26. 8% 38. 3% 56. 8% 33. 0% ns <. 001 <. 05 ns Pretest Content Exposure Exposed Content

Agent Evaluation Freudbot How enjoyable was this activity? How engaging was this activity? Would you recommend this activity to others? Overall, how would you rate this activity? How useful is this interface for learning information about Freud/Piaget? n=53 2. 92 3. 08 Piagetbot n=31 p 2. 87 ns 3. 19 ns 2. 83 3. 02 2. 97 2. 87 ns ns 2. 96 3. 42 ns Table 2: Ratings based on a 5 -point scale where higher numbers reflect positive experiences. Note: All t’s were non-significant

Discussion - the text-based interface was rated more positively and associated with better learning outcomes Limitations - orienting task biased towards text-based interface - degree of animation

Freudbot AIML Narrative format Speech Act theory

Freudbot findings

Conversational agents as pedagogical agents