Speech Generation From Concept and from Text Julia

Speech Generation: From Concept and from Text Julia Hirschberg CS 6998 11/23/2020 1

Today z TTS z CTS 11/23/2020 2

Traditional TTS Systems z Monologue y. News articles, email, books, phone directories y. Input: plain text y. How to infer intention behind text? 11/23/2020 3

Human Speech Production Levels z World Knowledge z Semantics z Syntax z Word z Phonology z Motor Commands, articulator movements, F 0, amplitude, duration z Acoustics 11/23/2020 4

TTS Production Levels: Back End and Front End z. Orthographic input: The children read to Dr. Smith z. World Knowledge text normalization z. Semantics z. Syntax word pronunciation z. Word z. Phonology intonation assigment y. F 0, amplitude, duration z. Acoustics synthesis 11/23/2020 5

Text Normalization z Context independent: y. Mr. , 22, $N, NAACP, MAACO VISA z Context-dependent: y. Dr. , St. , 1997, 3/16 z Abbreviation ambiguities: How to resolve? y. Application restrictions – all names? y. Rule or corpus-based decision procedure (Sproat et al ‘ 01) 11/23/2020 6

z Part-of-speech ambiguity: y. The convict went to jail/They will convict him y. Said said hello y. They read books/They will read books y. Use: local lexical context, pos tagger, parser? Sense ambiguity: I fish for bass/I play the bass Use: decision lists (Yarowsky ’ 94) 11/23/2020 7

Word Pronunciation z Letter-to-Sound rules vs. large dictionary y. O: _{C}e$ /o/ hope y. O /a/ hop z Morphological analysis y. Popemobile y. Hoped z Ethnic classification y. Fujisaki, Infiniti 11/23/2020 8

z Rhyming by analogy y. Meronymy/metonymy z Exception Dictionary y. Beethoven z Goal: phonemes+syllabification+lexical stress y. Context-dependent too: x. Give the book to John. x. To John I said nothing. 11/23/2020 9

Intonation Assignment: Phrasing z Traditional: hand-built rules y. Punctuation 234 -5682 y. Context/function word: no breaks after function word He went to dinner y. Parse? She favors the nuts and bolts approach z Current: statistical analysis of large labeled corpus y. Punctuation, pos window, utt length, … 11/23/2020 10

Intonation Assignment: Accent z Hand-built rules y. Function/content distinction He went out the back door/He threw out the trash y. Complex nominals: x. Main Street/Park Avenue xcity hall parking lot z Statistical procedures trained on large corpora z Contrastive stress, given/new distinction? 11/23/2020 11

Intonation Assignment: Contours z Simple rules y‘. ’ = declarative contour y‘? ’ = yes-no-question contour unless wh-word present at/near front of sentence x. Well, how did he do it? And what do you know? 11/23/2020 12

The TTS Front End Today z Corpus-based statistical methods instead of hand-built rule-sets z Dictionaries instead of rules (but fall-back to rules) z Modest attempts to infer contrast, given/new z Text analysis tools: pos tagger, morphological analyzer, little parsing 11/23/2020 13

TTS Back End: Phonology to Acoustics z Goal: y. Produce a phonological representation from segmentals (phonemes) and suprasegmentals (accent and phrasing assignment) y. Convert to an acoustic signal (spectrum, pitch, duration, amplitude) z From phonetics to signal processing 11/23/2020 14

Phonological Modeling: Duration z How long should each phoneme be? y. Identify of context phonemes y. Position within syllable and # syllables y. Phrasing y. Stress y. Speaking rate 11/23/2020 15

Phonological Modeling: Pitch z How to create F 0 contour from accent/phrasing/contour assignment plus duration assignment and phonemes? y. Contour or target models for accents, phrase boundaries y. Rules to align phoneme string and smooth y. How does F 0 align with different phonemes? 11/23/2020 16

Phonetic Component: Segmentals z Phonemes have different acoustic realizations depending on nearby phonemes, stress y. To/to, butter/tail z Approaches: y. Articulatory synthesis y. Formant synthesis y. Concatenative synthesis x. Diphone or unit selection 11/23/2020 17

Articulatory Synthesis-by-Rule z Model articulators: tongue body, tip, jaw, lips, velum, vocal folds z Rules control timing of movements of each articulator z Easy to model coarticulation since articulators modeled separately z But: sounds very unnatural y. Transform from vocal tract to acoustics not well understood y. Knowledge of articulator control rules incomplete 11/23/2020 18

Formant (Acoustic) Synthesis by Rule z Model of acoustic parameters: y. Formant frequencies, bandwidths, amplitude of voicing, aspiration… z Phonemes have target values for parameters z Given a phonemic transcription of the input: y. Rules select sequence of targets y. Other rules determine duration of target values and transitions between 11/23/2020 19

z Speech quality not natural y. Acoustic model incomplete y. Human knowledge of linguistic and acoustic control rules incomplete 11/23/2020 20

Concatenative Synthesis z Pre-recorded human speech y. Cut up into units, code, store (indexed) y. Diphones typical z Given a phonemic transcription y. Rules select unit sequence y. Rules concatenate units based on some selection criteria y. Rules modify duration, amplitude, pitch, source – and smooth spectrum across junctures 11/23/2020 21

Issues z Speech quality varies based on y. Size and number of units (coverage) y. Rules y. Speech coding method used to decompose acoustic signal into spectral, F 0, amplitude parameters y. How much the signal must be modified to produce the output 11/23/2020 22

Coding Methods z LPC: Linear Predictive Coding y. Decompose waveform into vocal tract/formant frequencies, F 0, amplitude: simple model of glottal excitation y. Robotic y. More elaborate variants (MPLPC, RELP) less robotic but distortions when change in F 0, duration z PSOLA (pitch synchronous overlap/add): y. No waveform decomposition 11/23/2020 23

y. Delete/repeat pitch periods to change duration y. Overlap pitch periods to change F 0 y. Distortion if large F 0, durational change y. Sensitive to definition of pitch periods z No coding (use natural speech) y. Avoid distortions of coding methods y. But how to change duration, F 0, amplitude? 11/23/2020 24

Corpus-based Unit Selection z Units determined case-by-case from large hand or automatically labeled corpus z Amount of concatenation depends on input and corpus z Algorithms for determining best units to use y. Longest match to phonemes in input y. Spectral distance measures y. Matching prosodic, amplitude, durational features? ? ? 11/23/2020 25

TTS Back End: Summary z Speech most natural when least signal processing: corpus-based unit selection and no coding…. but…. 11/23/2020 26

TTS: Where are we now? z Natural sounding speech for some utterances y. Where good match between input and database z Still…hard to vary prosodic features and retain naturalness y. Yes-no questions: Do you want to fly first class? z Context-dependent variation still hard to infer from text and hard to realize naturally: 11/23/2020 27

y. Appropriate contours from text y. Emphasis, de-emphasis to convey focus, given/new distinction: I own a cat. Or, rather, my cat owns me. y. Variation in pitch range, rate, pausal duration to convey topic structure z Characteristics of ‘emotional speech’ little understood, so hard to convey: …a voice that sounds friendly, sympathetic, authoritative…. z How to mimic real voices? 11/23/2020 28

TTS vs. CTS z Decisions in Text-to-Speech (TTS) depend on syntax, information status, topic structure, … information explicitly available to NLG z Concept-to-Speech (CTS) systems should be able to specify “better” prosody: the system knows what it wants to say and can specify how z But…. generating prosody for CTS isn’t so easy 11/23/2020 29

Next Week z Read z Discussion questions z Write an outline of your class project and what you’ve done so far 11/23/2020 30