Embodied Conversational Agents A Case Study of Freudbot

Embodied Conversational Agents: A Case Study of Freudbot Bob Heller, Ph. D Athabasca University November 3, 2004

Acknowledgements Mike Proctor – AIML programmer Dean Mah – Web implementation Billy Cheung – Graphics, test chatter Lisa Jewell – Chat log analysis, content developer, test chatter Julianna Charchun – Chat log analysis Jude Onuh – AIML programmer

Embodied Conversational Agents Definitions • Embodiment in Conversational Interfaces: REA (Cassel et al. , 1999) • Embodied Conversational Agents (Cassel, Sullivan, Prevost, & Churchill, 2000) – FMTB model Vos (2002) offers 5 features of ECA – – – Human like appearance Body used for communication purposes Natural communication protocols Multimodality Social role

Embodied Conversational Agents Anthropomorphic Agents Animated Interface Agents Animated Pedagogical agents Pedagogical Agent Persona Intelligent Tutoring Systems - Auto. Tutor (Graesser et al) http: //www. autotutor. org/index. htm Chatterbots or Chatbots - Weizenbaum’s (1966) Eliza

Embodied Conversational Agents - Why? primacy of conversation Constructivist theory The Media Equation Persona effect cognitive load

Embodied Conversational Agents Richard Wallace and A. L. I. C. E. • Artificial Linguistic Internet Computer Entity http: //alicebot. org/ • 3 time winner of the Loebner Contest (the holy grail for chatbots) http: //www. loebner. net/ • AIML – Artificial Intelligence Markup Language http: //www. aimlbots. com/ • Pandora. Bots http: //www. pandorabots. com

Embodied Conversational Agents ‘Theory’ behind ALICE - pattern matching - Zipf distribution - Iterative

Freudbot 1 Why Freud? • Initial plan of deployment • The famous personality application – Emile http: //www. hud. ac. uk/hhs/research/emileframeset. htm – Shakespeare http: //www. pandorabots. com/pandora/talk? botid=c 6937 cfb 3 e 354738 – Hans Christian Anderson http: //www. niceproject. com/about/ – John Lennon

Freudbot 1 Developing the AIML • Narrative structure • Test chatters • How much ALICE?

Freudbot 1 Research Questions • Is it worth it? • Is ‘chattiness’ related to the subjective evaluation of chat experience? • Are there individual difference variables that are related to measures of chat performance/experience?

Freudbot 1: Methodology • Online Recruitment – restricted to psychology students – Incentive (1/30 chance at $300) • Random assignment to bot type • Controlled Chat – automatically directed to questionnaire after 10 mins of chat

Freudbot 1: Participants (N=67) Gender Men Women n 12 55 Percent 18% 82% Age Distribution 18 -22 23 -27 28 -32 33 -37 38 -42 42+ 6 15 11 7 15 13 9% 22% 16% 10% 22% 19% Student Status Full-time Part-time Non-student 27 35 5 40% 52% 8% Self-rated academic ability Below avg Average Above avg Excellent 0 13 39 15 0% 19% 58% 22%

Is it worth it? • self-report data* Useful Recommend Overall Mean 2. 2 2. 4 Would you chat again? Yes No (n=30) (n=35) 2. 7 1. 8 3. 4 1. 6 3. 2 1. 8 Enjoyable Engaging Memorable 2. 6 2. 7 2. 8 3. 4 3. 6 1. 9 2. 1 2. 2 Expansion 3. 4 4. 1 2. 8 * 5 point scale

Is it worth it? Best Features Interactivity Able to ask questions with answers Learning about Freud & theories Simplicity/ease of use Entertaining/humorous Thought provoking No good features Technological features of Freudbot Potential to Freudbot Alternative learning style Novelty/uniqueness of Freudbot Tricking Freudbot Unpredictable Worst Features 16 16 13 5 5 4 4 3 3 2 2 Repetition Unable to answer questions Conversation did not flow Limited knowledge base User needed prior knowledge User was uncertain about what to do Not an effective learning tool Conversation was too short No sound 33 23 12 10 3 3 3 1 1

Is it worth it? • Chat logs Number of Exchanges Mean 31. 0 Range 5 -82 Proportion of on-task responses by participant* questions. 37 comments*. 23 Mean. 60 * correlated with a composite measure of self rated chat experience Proportion of repetitions by Freudbot Proportion of non-sensical by Freudbot . 25. 39

Chattiness? Freud. Alice Useful Recommend Overall Enjoyable Memorable Engaging Expansion # of Exchanges On task Response* n=35 2. 2 2. 5 2. 7 3. 0 2. 8 3. 3 32. 2. 56 * -significant difference btw groups Just. Freud n=32 2. 3 2. 4 2. 6 2. 7 3. 5 29. 7. 64

Individual difference variables? • demographic – – Gender Age Student status* Self-rated academic ability • computer experience & self-rated skill • academic background – – # of university courses # of distance ed courses* # of psychology courses Rated importance of Freud*

Individual difference variables? • attitudes towards technology and education – Positive aspects of on-line activities – Independent Learner – negative aspects of on-line activities*

Freudbot 1 Summary • Is it worth it? – worth another look • Is ‘chattiness’ related to the subjective evaluation of chat experience? – ‘Chattiness’ is not the right level – Nass and Reeves (1998) • Are there individual difference variables that are related to measures of chat performance/experience? – some relations that make sense and others that don’t

Freudbot 2 Research Goals 1. Improve Performance • Fix repetition problem • Topic tags • More content 2. Replication 3. Instructional Set 4. Future Development

Freudbot 2: Methodology http: //psych. athabascau. ca/html/Freudbot/test. html • online recruitment, incentive, & controlled chat identical to Freudbot 1 • random assignment to instructional set • similar questionnaire with additional questions on applications and improvements

Participants (N=55) Gender Men Women n 10 45 Percent 18% 82% Age Distribution 18 -22 23 -27 28 -32 33 -37 38 -42 42+ 7 17 7 11 6 7 13% 31% 13% 20% 11% 13% Student Status Full-time Part-time Non-student 26 28 1 47% 51% 2% Self-rated academic ability 0 -50 50 -65 66 -79 80 -89 90+ 0 2 11 30 10 4% 4% 20% 55% 18%

Improvement? • self-report data (5 point scale) Freudbot 1 Useful** 2. 2 Recommend** 2. 4 Overall** 2. 4 Freudbot 2 3. 0 2. 9 3. 0 Would you chat again? Yes No (n=37) (n=18) 3. 3 2. 4 3. 4 1. 7 3. 4 2. 2 Enjoyable Engaging** Memorable 2. 6 2. 7 2. 8 3. 0 3. 1 3. 3 3. 5 3. 6 2. 3 2. 2 2. 1 Expansion** 3. 4 4. 1 4. 4 3. 3 ** - statisically significant

Improvement? • Chat logs Number of Exchanges Mean 28. 4 Range 3 -115 Mean Proportion of on-task responses by participant*. 90 questions. 36 comments. 48 * correlated with a composite measure of self rated chat experience Proportion of appropriate responses by Freud . 60

Replication? • Demographic – – Gender* Age Student status* Self-rated academic ability • computer experience • academic background – – # of university courses # of distance ed courses # of psychology courses Rated importance of Freud*

Replication? • attitudes towards technology and education – Positive aspects of on-line activities – Independent Learner – negative aspects of on-line activities*

Instructional Set? Useful Recommend Overall Enjoyable Memorable Engaging Expansion Brief Set n=27 3. 1 2. 8 2. 9 3. 2 3. 0 3. 9 # of Exchanges On task Response 25. 3. 90 Elaborate Set n=28 2. 9 3. 1 3. 0 3. 3 4. 2 31. 3. 90

Future Development? Freudbot Improvements Other Applications Mean* Chat behaviour 4. 2 Audio Response 3. 1 Voice Recognition 2. 6 Synchronization 2. 5 Animation/movment 2. 3 * 5 -point scale Mean* Practice quizbot 4. 1 Famous personality 4. 1 Course content 3. 4 Chatroom 3. 3 Course Admin 3. 2

Freudbot 2: Summary 1. Improvement - yes, but clearly room for more 2. Replication - some 3. Instructional Set - no effects 4. Development

Future Direction • Haptek Freud – Animacy/agency hypothesis http: //psych. athabascau. ca/html/Freudbot/haptek. html • Piagetbot (Support from MCR) – learning outcomes • Skinnerbot (Lyle Grant) • Coursebot • Quizbot

Questions?