How to integrate automatic speech recognition ASR into

How to integrate automatic speech recognition (ASR) into CALL applications Helmer Strik Department of Linguistics Centre for Language and Speech Technology (CLST) Radboud University Nijmegen, The Netherlands Radboud University Nijmegen

Overview Introduction ASR: automatic speech recognition ASR-based tutoring ASR-based CALL ASR-based literacy training Conclusions LESLLA, Antwerpen, 24 -11 -2008 2 Radboud University Nijmegen

Introduction Students who receive 1 -on-1 instruction perform as well as the top two percent of students who receive traditional classroom instruction [Bloom 1984] A human tutor for every student is not feasible computer tutors For language learning: CALL Many text-based CALL systems Include speech-based CALL system LESLLA, Antwerpen, 24 -11 -2008 3 Radboud University Nijmegen

Speech inside Many applications with ‘speech’: Screen readers [#] Reading pen Mobile phone: photo + OCR + TTS Some also (useful) for CALL [#] LESLLA, Antwerpen, 24 -11 -2008 4 Radboud University Nijmegen

Speech inside (cont’d) Many applications with ‘speech’ Screen readers, reading pen, etc. Some also (useful) for CALL However, usually the learner can only listen (TTS: text-to-speech) or, also speak, but … no assessment, or the learner has to carry out the assessment, e. g. by comparing with examples use ASR / speech technology Is it feasible? LESLLA, Antwerpen, 24 -11 -2008 5 Radboud University Nijmegen

ASR: automatic speech recognition What is ASR? Speech to text conversion Applications: Dictation Command control Spoken dialogue systems (information) etc. ASR is not flawless, and it will probably never be esp. for non-native speech Note: this is not even the case for humans! LESLLA, Antwerpen, 24 -11 -2008 6 Radboud University Nijmegen

Speech Recognition mii nn cgn 2 -s vb LESLLA, Antwerpen, 24 -11 -2008 7 Radboud University Nijmegen

ASR-based tutoring ITS: Intelligent Tutoring Systems Spoken dialogue system for learning Subject matter: math, physics, etc. Examples: ITSPOKE, Univ. of Pittsburgh, Litman et al. Topic: Physics SCo. T, Stanford Univ. , Peters et al. Topic (SCo. T-DC): shipboard damage control Communicate with speech the subject matter doesn’t have to be speech LESLLA, Antwerpen, 24 -11 -2008 8 Radboud University Nijmegen

ASR-based CALL The subject matter is speech (language) Late 1990’s: 1998: STi. LL, Marholmen (Sweden); 1 st time the CALL and Speech communities met 1999: Special Issue of CALICO, 'Tutors that Listen‘, focusing on ASR (mainly ‘discrete ASR’) LESLLA, Antwerpen, 24 -11 -2008 9 Radboud University Nijmegen

ASR-based literacy training What has been done? Ø Reading tutors (the learner reads, not the PC): Listen, CMU, Pittsburgh; Mostow et al. (1994) STAR system, UK; Russel et al. (1996) SPACE, KU Leuven; Van hamme, Duchateau, et al. … and many others [#] Ø Ft. L: Foundations to Literacy, Boulder; Cole, Wise, et al. LESLLA, Antwerpen, 24 -11 -2008 10 Radboud University Nijmegen

ASR-based literacy training Foundations to Literacy Interactive Books Teach fluent reading & comprehension Foundational Skills Tutors Teach underlying reading skills Phonics LESLLA, Antwerpen, 24 -11 -2008 11 Radboud University Nijmegen

ASR-based literacy training (cont’d) What has been done? Reading tutors: Listen, CMU, Pittsburgh; Mostow et al. (1994) STAR system, UK; Russel et al. (1996) SPACE, KU Leuven; Van hamme, Duchateau, et al. …, and many others Ft. L: Foundations to Literacy, Boulder; Cole, Wise, et al. Mostly for children And for adults? What is needed? What is possible, and what is not? … LESLLA, Antwerpen, 24 -11 -2008 12 Radboud University Nijmegen

ASR-based CALL ASR is not flawless, and it will probably never be esp. for non-native speech Be aware of what is (not) possible with ASR technology Problematic issues and possible solutions: Noise, esp. background speech min. , head-sets Disfluencies min. , improve autom. handling Non-native pronunciation Recognizing utterances utterance verification Detect pronunciation errors classifiers LESLLA, Antwerpen, 24 -11 -2008 13 Radboud University Nijmegen

ASR-based CALL Our research: Non-natives Assessment of oral proficiency Dutch-CAPT – pronunciation o ASR / UV – Utterance Verification o PED – Pronunciation Error Detection DISCO – pronunciation, morphology, syntax TST-AAP People with speech disability for training & as communication aid (AAC) ASR for dysarthric speech EST: E-learning based Speech Therapy LESLLA, Antwerpen, 24 -11 -2008 14 Radboud University Nijmegen

ASR-based CALL Project Dutch-CAPT (Computer Assisted Pronuciation Training) LESLLA, Antwerpen, 24 -11 -2008 15 Radboud University Nijmegen

LESLLA, Antwerpen, 24 -11 -2008 16 Radboud University Nijmegen

ASR-based CALL (cont’d) Project Dutch-CAPT (CAPT: Computer Assisted Pronuciation Training) Exp. group: used the Dutch-CAPT system 2 control groups: didn’t use Dutch-CAPT The reduction in the number of pronunciation errors made was significantly larger for the exp. group, Training: 4 weeks x 1 session of 30’ – 60’ LESLLA, Antwerpen, 24 -11 -2008 17 Radboud University Nijmegen

ASR-based CALL (cont’d) ASR is not flawless, and it will probably never be esp. for non-native speech Be aware of what is (not) possible with ASR technology Problematic issues and possible solutions: Noise, esp. background speech min. , head-sets Disfluencies min. , improve autom. handling Non-native pronunciation Recognizing utterances utterance verification Detect pronunciation errors classifiers Mix of expertise needed: ASR techn. , L-acq. , pedagogy, design, … LESLLA, Antwerpen, 24 -11 -2008 18 Radboud University Nijmegen

ASR-based literacy training Demonstration project TST-AAP Existing course Add speech technology: Detect whether words & sounds were pronounced (correctly) LESLLA, Antwerpen, 24 -11 -2008 19 Radboud University Nijmegen

ASR-based literacy training Listening; PC: produces speech Text-To-Speech (TTS); quality good enough? Recorded speech, concatenation Speaking; PC: recognizes speech Phonics (see Ft. L) PC: Recognize words, utterances: CMs for Utt. Ver. PC: Recognize sounds: CMs for Phon. Ver. (contrasts) Reading (reading tutors) PC: Recognize words, utterances PC: Pointer in the text (‘track’ the reader) PC: Help when encountering problems PC: Change tempo read faster LESLLA, Antwerpen, 24 -11 -2008 20 Radboud University Nijmegen

ASR-based CALL Advantages of using speech (vs. writing) Self-explanation Extra information: Prosody (stress, accent) Emotions Confidence Other useful techniques: VTH [#] LESLLA, Antwerpen, 24 -11 -2008 21 Radboud University Nijmegen

Conclusions ASR is not flawless ASR-based tutoring is possible (restricted domain) general topics; ITS: ITSPOKE, SCo. T CALL; many systems: non-natives, disabled, etc. Literacy training So far mainly for children And for adults !? Needed Mix of expertise: techn. , L-acq. , pedagogy, design, … Improved ASR, speech technology Projects, funds LESLLA, Antwerpen, 24 -11 -2008 22 Radboud University Nijmegen

Questions? Why are there so few ASR-based CALL / literacy applications for adults? What are, in this context, important differences between children & adults? What is needed? Listening; Speaking; PC: produces speech PC: recognizes speech Phonics Reading (reading tutors) What else? LESLLA, Antwerpen, 24 -11 -2008 23 Radboud University Nijmegen

Questions? Why are there so few ASR-based CALL / literacy applications for adults? What are, in this context, important differences between children & adults? What is needed? Listening; Speaking; PC: produces speech PC: recognizes speech Phonics Reading (reading tutors) What else? LESLLA, Antwerpen, 24 -11 -2008 24 Radboud University Nijmegen

LESLLA, Antwerpen, 24 -11 -2008 25 Radboud University Nijmegen