Acoustic Databases Jan Odijk ELSNET Summer School Prague

Acoustic Databases Jan Odijk ELSNET Summer School, Prague, 2001

Acknowledgements n Part of the slides n have been borrowed from or n are based on work by • • • Bart D’Hoore Hugo van Hamme Robrecht Comeyne Dirk van Compernolle Bert van Coile

Overview q What is a speech database? q How is it used? q What does it contain? q How is it created? q Industrial needs q Technologies and applications

Overview Ø What is a speech database? q How is it used? q What does it contain? q How is it created? q Industrial needs q Technologies and applications

Linguistic Resources(LRs) n Linguistic Resources are sets of language data in machine readable form that can be used for developing, improving or evaluating language and speech technologies. n Some language and speech technologies • • • Text-To-Speech (TTS) Automatic Speech Recognition (ASR) Dictation Speaker Verification/recognition Spoken Dialogue Audio Mining Machine Translation Intelligent Content Management ….

Linguistic Resources(LRs) Major Types n Electronic Text Corpora • • • Newspapers, magazines, etc. Usenet texts, e-mail, correspondence Etc. n Lexical Resources • • Monolingual lexicons Translation lexicons Thesauri … n Acoustic Resources • • Annotated Speech Recordings Annotated Recordings of other acoustic signals • Coughing, throat clearing, breathing, … • Door slamming, screeching tires (of a car), …

Types of Linguistic Resources Acoustic Resources n Acoustic Databases (ADBs) • • • Controlled recording of human speech or other acoustic signals Enriched with annotations Recorded in a digital way Representative of targeted application environment and medium Balanced for phonemes/phoneme combinations Speaker parameters, recording quality, environment/medium documented

Types of Linguistic Resources Acoustic Resources n Annotated unstructured recordings • • Broadcasted material Recorded conversations/monologues/speeches etc Dictated material Enriched with annotations

Types of Linguistic Resources Acoustic Resources n In-service data • • Recorded sessions of interaction humans-running application Usually by logging a customer system Enriched with annotations Used for tuning models, grammars, etc. to specific application

Types of Linguistic Resources Acoustic Resources n Environments • “Quiet” • Studio • Quiet office • Normal office • Noisy • Public place (street, hotel lobby, station, etc. ) • Car (running engine 0 km/hr, city, highway) • Industrial environment

Types of Linguistic Resources Acoustic Resources n Media • • • HQ close-talk microphone Desktop Microphones Telephone • analog or digital • fixed line or mobile • • • Wide band microphones Array microphones PC/PDA etc. low quality microphone

Overview q What is a speech database? Ø How is it used? q What does it contain? q How is it created? q Industrial needs q Technologies and applications

Acoustic Resources Use n (for speech synthesis modules in TTS systems) n (as acoustic reference material for pronunciation lexicons) n Mainly for speech recognition n Training and test material for research into new recognition engines and engine features n Training and test material for development of acoustic models n Tuning of acoustic models for specific applications

What is speech recognition? n ASR: Automatic speech recognition n Automatic speech recognition is the process by which a computer maps an acoustic speech signal to text. n Automatic speech understanding is the process by which a computer maps an acoustic speech signal to some form of abstract meaning of the speech. n Speaker recognition is the process by which a computer recognizes the identity of the speaker based on speech samples. n Speaker verification is the process by which a computer checks the claimed identity of the speaker based on speech samples.

Elements of a Recognizer

Elements of a Recognizer Acoustic Model Speech Data Feature Extraction Pattern Matching Language Model Action Post Processing Natural Language Understanding Display text Meaning

Feature Extraction n Turning speech signal into something more manageable • • Do analysis once every 10 ms Data compression: 220 byte => 50 byte => 4 byte n Sampling of a signal: transforming into a digital form n Extracting relevant parameters from the signal • Spectral information, energy, pitch, . . . n Eliminate undesirable elements (normalization) • • • Noise Channel properties Speaker properties (gender)

Feature Extraction: Vectors n Signal is chopped in small pieces (frames), typically 30 ms n Spectral analysis of a speech frame produces a vector representing the signal properties. n => result = stream of vectors 10. 3 1. 2 -0. 9. 0. 2

Elements of a Recognizer Acoustic Model Speech Data Feature Extraction Pattern Matching Language Model Action Post Processing Natural Language Understanding Display text Meaning

Acoustic Model n Split utterance into basic units, e. g. phonemes n The acoustic model describes the typical spectral shape (or typical vectors) for each unit n For each incoming speech segment, the acoustic model will tell us how well (or how badly) it matches each phoneme n Must cope with pronunciation variability • • • Utterances of the same word by the same speaker are never identical Differences between speakers Identical phonemes sound differently in different words => statistical techniques: creation via a lot of examples

f-r--ie--n--d--l--y- c--o--m--p---u----t--e--r---s S 1 S 2 S 3 S 4 S 5 S 6 S 7 S 8 S 9 S 10 S 11 S 12 S 13

Acoustic Model: Units n Phoneme: share units that model the same sound S T O P S T A R Stop T Start n Word: series of units specific to the word S 1 S 2 S 3 S 4 Stop S 6 S 7 S 8 S 9 S 10 Start

Acoustic Model: Units n Context dependent phoneme S|, |T T|S|O O|T|P P|O|, ST TO OP Stop n Diphone , S P, Stop n Other sub-word units: consonant clusters ST O P Stop

Acoustic Model: Units n Phonemes in context: spectral properties depend on previous and following phoneme n Diphones n Sub-words: syllables, consonant clusters n Words n Multi words: example: “it is”, “going to” n Combinations of all of the above

Elements of a Recognizer Acoustic Model Speech Data Feature Extraction Pattern Matching Language Model Action Post Processing Natural Language Understanding Display text Meaning

Pattern matching n Acoustic Model: returns a score for each incoming feature vector indicating how well the feature corresponds to the model. = Local score n Calculate score of a word, indicating how well the word matches the string of incoming features (viterbi) n Search algorithm: looks for the best scoring word or word sequence

Elements of a Recognizer Acoustic Model Speech Data Feature Extraction Pattern Matching Language Model Action Post Processing Natural Language Understanding Display text Meaning

Language Model n Describes how words are connected to form a sentence n Limit possible word sequences n Reduce number of recognition errors by eliminating unlikely sequences n Increase speed of recognizer => real time implementations

Language Model n Two major types • • Grammar based !start <sentence>; <sentence>: <yes> | <no>; <yes>: yes | yep | yes please ; <no>: no | no thanks | no thank you ; Statistical • Probability of single words, 2/3 -word sequences • Derived from frequencies in a large corpus

Active Vocabulary n Lists words that can be recognized by the acoustic model n That are allowed to occur given the language model n Each word associated with a phonetic transcription • • Enumerated, and/or Generated by a Grapheme-to-Phoneme (G 2 P) module

Post Processing n Re-ordering of Nbest list using other criteria: e. g. account numbers, telephone numbers n Spelling: name search from a list of known names n Applying NLP techniques that fall outside the scope of the statistical language model • • • E. g. “three dollars fifty cents” “$ 3. 50” “doctor Jones” “Dr. Jones” Etc.

Training of Acoustic Models Annotated Speech Database Pronunciation Dictionary Training Program Acoustic Model

Training of Acoustic Models n Database design • • • Coverage of units: word, phoneme, context dependent unit Coverage of population (region, dialect, age, …) Coverage of environments (car, telephone, office, . . ) n Database collection and validation • • Checking recording quality Annotation: describing what people said, extra-speech sounds n Dictionaries • • • Phonetic transcription of words Multiple transcriptions needed G 2 P: automatic transcription

Feature vectors . . . 10. 3 1. 2 -0. 9. 0. 2 8. 1 -0. 5 1. 3. 0. 2 ……. . . 2. 1 -0. 2 1. 9. -0. 3 . . .

Example: discrete models n A collection of prototypes is constructed (100 to 250) n Each vector is replaced by its nearest prototype

Feature vectors Prototypes . . . 10. 3 1. 2 -0. 9. 0. 2 8. 1 -0. 5 1. 3. 0. 2 ……. . . 2. 1 -0. 2 1. 9. -0. 3 . . . , , , 3 9 , , , , , , , , 7, , ,

Phoneme assignment ffrr. EEEnnnnddll. IIII, , , kk. OOOmmpjjuuuuttt$$$rrzz Prototypes 2276998900023448889211127780128897791237787622

Training of Acoustic Models For all utterances in database: Make phonetic transcription of a sentence Use models to segment the utterance file: assign a phoneme to each speech frame Collect statistical information: Count prototype-phoneme occurrences Create New Models

Key Element in ASR is based on learning from observations • • Huge amount of spoken data needed for making acoustic models Huge amount of text data needed for making language models n => Lots of statistics, few rules

Overview q What is a speech database? q How is it used? Ø What does it contain? q How is it created? q Industrial needs q Technologies and applications

Contents of an ADB q Utterances of different utterance types q Utterance types suited to the intended application domain q Text balanced for phoneme and/or diphone distribution q All enriched with annotations

Contents of an ADB Spontaneous v. Read Utterances 1. A spontaneous utterance is a response to a question or a request 1. 2. 3. 4. “In which city do you live? ” “Please spell a letter heading to your secretary” “Is English your mother tongue? ” “Make a hotel reservation” 2. A read utterance is an utterance read from a presentation text 1. 2. 3. 4. “London” “Dear John” “Yes” “Please book me a room for 2 persons with bath. We will arrive …. ”

Contents of an ADB q Isolated Phonetically Rich Word q Apple Tree, Lobster q Isolated Digit q 5 q Isolated Alphabet q B q Isolated number (natural number) q 4256

Contents of an ADB q Continuous Digits q 9 1 1 q Continuous Alphabet q Y M C A q Commands q Stop, left, print, call, next

Contents of an ADB Connected Digits q Telephone Numbers q 057/228888 q Credit Card Numbers q 3741 959289 310001 q Pin-codes q 8978 q Social Security Number q 560228 561 80 q Other identification numbers, e. g. sheet id q 012589225712

Contents of an ADB Time and Date Expressions q Time (“analog”, word style) q A quarter past two q Time (“digital”) q 14: 15 q 2: 15 PM q Date (“analog”, word style, absolute) q Friday, June 25 th, 1999 q Christmas’ Eve, Easter q Date (“digital”, absolute) q 25/06/99 q Date (“analog”, word style, relative) q Tomorrow, next week, in one month

Contents of an ADB q Money amounts q $327. 67 q £ 148. 95 q Isolated Phonetically Rich Sentences q A cold supper was ordered and a bottle of port q Isolated Command Sentences q Insert this name in the list q Names q Microsoft, New York, Jonathan q Syllables q Hi-ta-chi

Contents of an ADB q Continuous Phonetically Rich Sentences q Once upon a time, in a land far from here, lived a little princess. She was the most beautiful girl… q Continuous Command Sentences q Select the first line. Make it bold and move it to the bottom of the text… q Continuous Spontaneous speech q <Make a reservation in a hotel>

Contents of an ADB Contents of Speech. Dat-II ADB q For each speaker/session q q Approx. 40 utterances Duration approx. 10 minutes Mixture of read and spontaneous utterances Mixture of q Phonetically rich sentences q Application specific words q Utterance types that will often occur in any application q 1000 -5000 speakers/sessions

Contents of an ADB Contents of Speech. Dat-II ADB q 1 isolated digit q 4 -digit id (sheet number) q 3 connected digits (~10 -digit telephone number) q 12 -digit credit card number q 3 natural numbers q 2 money amounts: 1 large, 1 small q 3 spelled words q 1 time of day (spontaneous)

Contents of an ADB Contents of Speech. Dat-II ADB q 1 time phrase (read, word style) q 1 date (spontaneous, e. g. person’s birthday) q 2 dates (read, word style) q 3 yes/no questions q 1 city of call/birth q 6 common application words out of 50 q 3 application word phrases q 9 sentences (read)

Contents of an ADB Contents of Speech. Dat-Car ADB q For each session q q Approx. 120 -130 utterances (depending on session) Duration 2 -3 hours Mixture of read and spontaneous utterances Mixture of q Phonetically rich sentences q Application specific words q Utterance types that will often occur in any application q 600 sessions with min. 300 speakers q In 2 out of 7 conditions q Standing still/ Low speed/ high speed q Different road conditions/ surrounding noise q Audio equipment on/off

Contents of an ADB Contents of Speech. Dat-Car ADB q Digits and Digit Strings q q q 1 sequence of 10 digits 1 sheet number (4+ digit sequence) 1 spontaneous telephone number 1 credit card number (16 digits) 1 PIN code (6 digits) 4 isolated digits q Dates q 1 spontaneous date (e. g. birthday) q 1 prompted date, word style q 1 relative and general date expression

Contents of an ADB Contents of Speech. Dat-Car ADB q Names q q q 1 spontaneous name (e. g. speaker’s first name) 1 city of growing up (spontaneous) 2 most frequent city’s 2 company / agency / street names 1 person name (first name or surname) q Spellings q q 1 spontaneous spelled name (e. g. speaker’s first name) 1 spelling of city name 4 real words or names 1 artificial name (for coverage)

Contents of an ADB Contents of Speech. Dat-Car ADB q Money Amounts/Natural Numbers q 1 money amount q 1 natural number q Times q 1 time of day (spontaneous) q 1 time phrase (word style) q Phonetically Rich words q 4 phonetically rich words

Contents of an ADB Contents of Speech. Dat-Car ADB q Application words q q q 13 Mobile Phone Application words 22 IVR functions keywords 32 car products keywords 2 voice activation keywords 2 language dependent keywords q Sentences q 2 phrases using an application word q 9 phonetically rich sentences q 10 prompts for spontaneous speech

Overview q What is a speech database? q How is it used? q What does it contain? Ø How is it created? q Industrial needs q Technologies and applications

Phases in Acoustic Resource Creation n Design n Creation of Script n Recruitment & Recording n Annotation and Validation n Lexicon n Quality control n Production

DESIGN n Language study: • Phoneme set • Dialects n Scripting: utterance definition and distribution over speakers n Speaker typology: distribution definition • Sex, gender, age, dialects, educational level n Recording: specification of procedure and platform n Validation: specification of procedure and quality standard

Creation of Script Prompts/Text/Transcription n A prompt refers to the way an utterance is presented to the speaker. This can be done on the desktop, on paper or with a play back file (telephony). n The (presentation) text represents the utterance as it should be pronounced by the speaker. It is normally presented according to the spelling conventions of the target language. n The transcription is the utterance as it has been pronounced by the speaker. n EXAMPLE: The pronunciation of a digit string • • • n PROMPT: “Please read the number on top of your form” TEXT: 578124 TRANSCRIPTION: five seven eight one two four

Creation of Script n Collect and clean text corpora § Split cleaned text into a sequence of sentences § Remove ungrammatical and too long sentences § Remove sentences containing offensive language § Remove (certain) ambiguities in pronunciation § numbers, dates, abbreviations, etc. § Apply phonetic balancing tools to obtain phonetically rich text

Creation of Script n Collect and/or create other utterance types • Telephone numbers, amounts, credit card numbers, etc. n Create prompts • Prerecorded messages to the speaker • For unmonitored recording without access to screen (telephony) n Put all these in resource files

Creation of Script File n Configuration: • Acquisition board, Coding type • Sampling rate, Number of channels n Information items • Speaker id, Sheet id • Gender, age, region of birth, region youth, living, etc. and their possible values • Recording environment/conditions n Sentence definitions • Specifies order and types of utterances in one session

Creation of Script n Resource files utterance sheets n Generate letter with instructions and list of utterances for each speaker (esp. telephony)

Creation of Script Tools n Script Editor • For creating/modifying scripts • For creating utterance sheet files (from resource files) • For generating letters to speakers n Digit String Generator • Natural numbers • Bank accounts • Credit card numbers • Phone numbers • Pin-codes

Creation of Script n Test the script • By making one or more recording sessions • Also tests the recording set-up • also provides indications for average session duration

RECRUITMENT n Contact potential speakers according to the typology • Acquaintances, colleagues • Advertisements • Employees/students of cooperating organizations (companies, universities) • Possibly with the help of marketing agencies n Explain • purpose and context • What the speaker is supposed to do • How much time it will take • Reimbursement for the speaker (time spent, travel costs) n Make concrete arrangements with the speakers

RECORDING n Locations : Set up recording equipment in environment fitting environment definition n Set-up recording platform and test it n Welcome speaker, instruct speaker n Interview : log speaker typology & recording conditions n Make recordings and follow-up on quality n Deal with administrative matters • Agreement on ownership of recording • Reimbursement

RECORDING TOOL

VALIDATION and ANNOTATION n After recording, the signal will NEVER be touched • Only enriched with annotations n check (and correct) relation between text & speech • Orthographic transcription must represent what the speaker said • Tool to expand abbreviations, numbers, digit sequences n Segmentation • Check (and correct) begin and end of speech markers • (mostly for TTS) Mark begin and end of phonemes

VALIDATION and ANNOTATION n assign quality label • Very good overall quality … very bad overall quality n Annotations for extra events • Speaker sounds (coughing, breathing, swallowing, …) • Mispronunciations, truncations • Sound from other sources (other speaker, music, radio, …) • Continuous background noise (wind, rain, …) • Filled pauses (uh, um, er, ah, hmm, …. ) • Telephone distortions

Validation Tool

Semi-Automatic Validation n Validation can be partially automated n For certain types of databases n 70 -75% reliably validated automatically n 25 -30% require manual check n Using ASR systems n Research into further automating this ongoing

LEXICON n One central “mother lexicon” for each language • To reduce duplication of effort • To maintain consistency n Request is compared with mother database • Unfound entries imported in the mother database • Unfound entries turned into a job • Job is assigned to linguists

LEXICON n After finishing job • Requested entries and properties are exported • Turned into required format • Delivered to requestor n Additions/modifications due to this request are now available for other requests

LEXICON: Tools n Phoned • Lexical database plus user interface • (currently in Access but switching to SQL Server) • Reuse of G 2 P and Synthesis Modules n Phoned. Admin • import and export of data from the mother database • Comparison with existing mother database • Definition of users and jobs • Assignment of jobs to users

QUALITY CONTROL o Typical Circumstances o o o Database project is ongoing Often on a remote location Multiple persons (for recording and validation) Many questions, problems and unclarities arise constantly Require answers from specialists Danger of errors and inconsistencies o Within the work of a single person o between different persons o Constant monitoring o Systematic and regular quality checks required o Systematic and regular feedback required o During the whole project o From the earliest moment possible n Documentation, incl. spot check report

QUALITY CONTROL n Tools • ADB Scanner—checks consistency of database • Standard structure, All files available • ADB Statistics • Statistics on information items (sex, gender, age, dialect, quality, etc. ) and utterance types • ADB Report Tool • For creating parts of the documentation • And others

PRODUCTION n Huge amount of data! n Multiple copies needed n Special fast CD-replicator equipment n Special cupboards for storing the CDs n Description in catalogue n Distribution n Conversion Tools (format converter, down sampling, demultiplexing)

DAR Resource Description n

DAR Resource Description: Statistics

Overview q What is a speech database? q How is it used? q What does it contain? q How is it created? Ø Industrial needs q Technologies and applications

General q More data! q The right data! q High Quality data q In-service Data q ASAP

Speech. Dat Family q Consortium of industrial and university partners q Often EU projects q One type of database is defined q Each partner makes one database according to spec q Each database is validated by external organization (SPEX, Nijmegen, the Netherlands) q After approval databases are exchanged among the partners q Max. 1 -1. 5 yr later data are offered for distribution by ELRA q http: //www. speechdat. org/

Overview of major projects q Speech. Dat (M) q Speech. Dat-II q Speech. Dat-E q Speech. Dat-Car q SPEECON q SALA II

Speech. Dat (M) q EU-funded q production, standardization, evaluation and dissemination of Spoken Language Resources q 8 fixed telephone network databases, 1000 speakers each; 1 mobile telephone network database, 300 speakers q Period: 1994 -1996

Speech. Dat (M) Partners q Siemens q UPC q Philips q IDIAP q Vocalis q INESC q CSELT q GEC MSIS

Speech. Dat (M) Languages q German q Spanish q French q Portuguese q Danish q Swiss French q Italian

Speech. Dat-II q EU-funded q Creation of Telephony Databases q 25 fixed and mobile telephone network databases, 500 -5000 speakers each; 3 speaker verification databases q Period: 1996 -1998

Speech. Dat-II Partners q Aalborg University q Auditex q British Telecom q CSELT q DMI q ELRA q GEC q GPT q IDIAP q INESC q SPEX q Knowledge S. A. q Swiss Telecom q KTH q Telenor q Lernout & Hauspie q Univ. of Maribor q Matra Nortel q Philips q Portugal Telecom q Siemens q Univ. of Munich q Univ. of Patras q UPC q Vocalis

Speech. Dat-II Languages q Danish q Finnish q Slovenian q Flemish q Finnish Swedish q Greek q Belgian French q Italian q Luxemburg German q Dutch q Portuguese q Swiss French q Spanish q Luxemburg French q Swiss German q Swedish q German q Norwegian q British English q Welsh

Speech. Dat-E q EU-funded q Eastern European Speech Databases for Creation of Voice Driven Teleservices q Speech databases for fixed telephone networks q suited for typical present-day teleservices plus q phonetically rich set of material for vocabulary independent ASR q 1000 – 2500 speakers q Period: 1999 -2001

Speech. Dat-E Partners q Auditex q Lernout & Hauspie q Philips Speech Processing q Siemens q ELRA q SPEX q Brno University of Technology q Prague Technical University q Budapest University of Technology q Wroclaw University of Technology q Slovak Academy of Sciences

Speech. Dat-E Languages q q q Russian (2500) Czech Slovak Hungarian Polish

Speech. Dat-Car q EU-funded q 9 in-vehicle and mobile telephone network databases q 300 speakers, each in 2 out of 7 conditions (600 recording sessions) q 5 simultaneous channels q Period: Apr 1998 - Oct 2000

Speech. Dat-Car Partners q Aalborg University q Nokia q Alcatel q Renault q Robert Bosch Gmb. H q SEAT q DMI q SPEX q ELRA q University of Munich q Knowledge S. A. q UPC q Lernout & Hauspie q L&H France (formerly Matra Nortel) q Vocalis q Volkswagen

Speech. Dat-Car Languages q Danish q German q British English q Greek q Finnish q Italian q Flemish/Dutch q Spanish q French q American English

SPEECON q Speech driven interfaces for consumer devices n Speech databases for voice controlled consumer applications • television sets, video recorders, mobile phones, palmtop computers, car navigation kits or even microwave ovens and toasters. n 600 speakers n Period: 2000 -2003

SPEECON Partners n Daimler. Chrysler n Nokia n Ericsson n Philips Speech Processing n IBM n Lernout & Hauspie n Natural Speech Communications n Siemens n Sony n Temic Telefunken

SPEECON Languages n EU Spanish n Flemish n Dutch n Russian n US English n Japanese n Italian n US Spanish n Polish n Swedish n German n UK English n Danish n Hebrew n Portuguese n French n Swiss German n Finnish n Cantonese n Mandarin

SALA I q Speech. Dat Across Latin America q Not government-subsidized q Speech databases for fixed telephony, Latin America q 1000 -2000 speakers per database q Period: 1998 -2001

SALA Partners q CSELT q Siemens q ELRA q SPEX q Lernout & Hauspie q UPC q Lucent q Vocalis q Philips

SALA Languages q Brasil (Portuguese, 2000) q Mexico (2000) q Caribbean islands and Venezuela q Panama, Columbia q Ecuador, Peru, Bolivia q Chile q Central America q Argentina, Uruguay, Paraguay

SALA II q Not government-subsidized q to create speech databases telephone cellular oriented applications q America (North and Latin) q 1000 (or 2000) speakers q Period: 2001 -2002 q (project just starting up)

SALA II Partners q ATLAS q NSC q ELRA q Philips q IBM q Siemens q Lernout&Hauspie q SPEX q Loquendo q UPC q Lucent

SALA II Languages q Venezuela q Peru q Mexico q Chile q Argentina q Costa Rica q Brasil q Colombia q American English Canada q US English North East q US Spanish East q US English South East q US Spanish West q US English North West

Future n Non-native/multilingual ASR n Data for Speech-to-Speech Translation n Access to information • • • anytime anywhere by way of any device n More use of spontaneous speech (“conversational systems”)

Future n Devices will become • • increasingly smaller (“mobile”) Increasingly more powerful Connected to information sources such as Internet etc robustness against different environments n Input/Output • • • Limited Keyboard and screens less convenient Opportunity for speech input and output Other input/output methods get different roles Multi-modal input and output systems

Future n Distributed systems • • • Part of the recognition/synthesis on the local system (“client”) Part on the server Dynamically adaptable local systems n In car, speech is • • “Hands-free” and “Eyes-free” solution

Overview q What is a speech database? q How is it used? q What does it contain? q How is it created? q Industrial needs Ø Technologies and applications

Why Speech User Interface n Pro • • • n Con Audio feedback draws attention Complex commands E. G. Control your VCR Fast and simple - Chinese !!! • • Speech input: 50 -250 wpm Typing: 20 -90 wpm Handwriting: 25 wpm Pointing: 10 -40/min Eyes free Hands fee Mobile Compact i/o devices • • • audio messages difficult to remember if too long. E. g. telephone number, address “a drawing can replace a thousand words” privacy sometimes cumbersome. E. g. control a cursor on a screen voice wear-out

Text-to-Speech engines Voice Quality Human-like Real. Speak Compact TTS 3000 Real. Speak Ultra. Comp act TTS 25 00 Machine-like low high Processor power & memory

Text-to-Speech engines n TTS 2500 • • Low quality, small footprint engine for talking dictionary products Available, no additional R&D n TTS 3000 • • • Medium quality engines Limited footprint, high densities Limited developments n Real. Speak compact • Target: handheld devices n Real. Speak • High-end system

Real. Speak TTS n New generation, human sounding TTS n Target: server based telephony, PCMM n Platform Requirements: • • CPU: 48 real time instances PIII 450 MHz (8 k. Hz speech data) RAM: < 250 k. B/instance ROM: 4 -6 MB n Speechbases: • • • 8 k. Hz uncompressed: ~ 250 MB 8, 11 k. Hz compressed: 20 – 30 MB 22 k. Hz compressed: 70 – 90 MB n 20 languages: US English, 15 European and 4 Asian languages n 2 languages under development

Real. Speak Compact n High quality, medium footprint TTS n Target: mobile and embedded platforms n Platform Requirements: • • • 150 MIPS RAM: < 250 k. B/instance; 4 -6 MB common ROM: 16 MB (includes 11 k. Hz speechbase) n Derived automatically from Real. Speak n Real. Speak ultra compact under development

TTS 3000 n Low footprint, highly intelligible TTS engine n Target: Telephony, PCMM, Mobile, Embedded n Platform requirements: • • CPU: 20 – 30 MIPS RAM: 100 k. B/instance; ROM: 2 - 3 MB n 13 languages including: • • • US English 7 European 3 Asian languages n 2 languages under development

TTS 2500 n Dedicated TTS for very low footprint talking dictionaries n Analysis on 8 or 16 bit processor: <2 Mips n Synthesis on dedicated chip (LH 3010 or LH 3030 ) or DSP (ADSP 21 xx) n 1. 5 MB ROM, 16 KB RAM n Languages: • • • American English Mandarin Chinese Mexican Spanish German French

Dimensions of ASR n Speaker • • • Independent - adaptive - dependent Native - non-native Man, woman, child n Recording conditions • • Recording device: telephone, GSM, microphone, tape recorder Environment: quiet office, home, car, factory, street… n Implementation • • Platform: PC, embedded CPU and memory

Dimensions of ASR n Size of the (active) vocabulary • Small (10 -100) - medium (100 -1000) - large(>1000) - very large (>10000) n Flexibility of the vocabulary • • • Fixed (factory-definable)- User-definable phoneme based => speaker independent user words => speaker dependent n Word sequences • • • Isolated words - sentences - word spotting Fixed grammar - flexible language model Discrete - continuous speech n Language • • Language independent engine, language dependent data files Swapping language files

Different Applications, Different Needs n Dictation • Speaker dependent, large vocabulary, continuous speech, quiet office, PC n Command & control, name dialing • Speaker independent, small to large vocabulary, noise robust, DSP boards and/or client-server n Dialogue systems • Speaker independent, medium to large vocabulary, noise robust, client-server n Security: verification • Speaker dependent, combination of password (what) + speaker characteristics (who) n Language learning • Non-native speakers, punish mistakes rather than being tolerant

Automatic Speech Recognition n L&H speech recognition engines cover a broad range of tasks, processor types, operating systems and input signal types: n Tasks: • • Large vocabulary continuous real-time dictation, Large vocabulary batch transcription, Grammar-based recognition – large, medium and small vocabularies, Small-vocabulary isolated word recognition. n Platforms: • • PC, Server, Handheld, Embedded, Distributed.

Automatic Speech Recognition engines Task Complexity Server Large Vocabulary Open Grammar Dictation Mobile Terminal XCalibur, MREC VX Large Vocabulary Closed Grammar ASR 1600 ASR 1500 Medium Vocabulary Closed Grammar ASR 300 Isolated word recognition ASR 100 low high Processor power & memory

Recognition engines … n Input conditions: • • Environments: home, office, public/industrial, car. Channels: telephone (wireline, wireless), wideband, mobile devices. Microphones: close-talking, far-talking. Combinations: e. g. , broadcast material. n A wide range of processor/memory operating points: • • 200 Mips/32 MB, 60 Mips/1 MB, 20 Mips/300 KB, 5 -10 Mips/<30 KB

Recognition engines …. n ASR 100: • • • 5 -10 Mips, < 30 KB, Speaker-dependent Recording device: mic. /phone Sampling Frequency: 8/11 k. Hz Environment: office Vocabulary: small and user-adaptable Grammar: isolated Speech: Isolated OS: various Architecture: stand-alone Languages: language-independent n Applications • • • embedded cell-phone dialing toys

Recognition engines …. n ASR 300: • • • • 20 Mips, 300 KB, SI & SD Sampling Frequency: 8/11 k. Hz Vocabulary: small and factory-adaptable Highly noise robust Environment: office/car/other noisy environments Unit: word-dependent Grammar: isolated Speech: quasi-connected command control OS: various Architecture: stand-alone Languages: US English, French, Italian, Korean, German, Japanese n Applications • • • In car command control Command control of toys, games Command control in noisy industrial environments

Recognition engines …. n ASR 1500 • • • • 60 Mips, 1 MB SI and speaker-adaptive Vocabulary: medium size; user-adaptable Sampling Frequency: 8 k. Hz Environment: office Recording Device: telephone/ mobile phone Grammar: finite state Speech: Continuous Unit: phoneme OS: various Architecture: Stand-alone and client/server Languages: US English, 10 European languages, 4 Asian languages n Applications • IVR applications over the phone • Reverse directory, Automated attendant • Information provider— stock quotes • Ordering systems

Recognition engines …. n ASR 1600 • • • • 60 Mips, 1 MB SI and speaker-adaptive Vocabulary: medium size; user-adaptable Sampling Frequency: 11 k. Hz Environment: office, car; highly noise-robust Recording Device: mic. Grammar: finite state Speech: Continuous Unit: phoneme OS: various Architecture: Stand-alone and client/server Languages: US English, 10 European languages, 4 Asian languages n Applications • In-Car recognition • Command Control • Embedded devices • PDA’s, Smart. Phones

Recognition engines …. n Mrec/VX: • • • • > 200 Mips > 64 MB SI and speaker-adaptive Vocabulary: very large (> 64, 000 words) Sampling Frequency: 22 (16) k. Hz Environment: Office Recording Device: mic. Grammar: statistical Speech: Continuous Unit: phoneme OS: Windows Architecture: Stand-alone Languages: US English and Spanish, 7 European languages, 2 Asian languages n Applications • • • document creation, incl. command control Media. Indexer (Mrec) Speech Transcription (Mrec)

Recognition engines …. n Xcalibur • • • scalable SI and speaker-adaptive Vocabulary: very large (> 64, 000 words) Sampling Frequency: 22 (16) k. Hz Environment: Office, (Telephony, Car) Recording Device: mic. Grammar: statistical and rule-based Speech: Continuous Unit: phoneme OS: Windows Architecture: Stand-alone and client-server Languages: Currently only Japanese n Applications • • • document creation command control Focus on conversational systems