Phoneme Alignment based on Discriminative Learning Shai ShalevShwartz

Phoneme Alignment based on Discriminative Learning Shai Shalev-Shwartz The Hebrew University, Jerusalem Joint work with Joseph Keshet, Hebrew University Yoram Singer, Google Dan Chazan, IBM Phoneme Alignment. Slide

The Alignment Problem Text: Have a test Phonetic transcription: /hh ae v ey tcl t eh s tcl t/ Waveform: Phoneme Alignment. Slide 2

The Alignment Problem Setting acoustic representation start-time of phoneme pi in x alignment function phonetic representation /hh ae v ey tcl t eh s tcl t/ Phoneme Alignment. Slide 3

Acoustic Representation Short-time Fourier Transform Phoneme Alignment. Slide 4

Comparing Alignments e. g. Phoneme Alignment. Slide 5

-insensitive Cost -insensitivity region Phoneme Alignment. Slide 6

A Discriminative Learning Approach Training set: Learning Algorithm Hypotheses class Alignment function: Phoneme Alignment. Slide 7

Outline of Solution 1. Define the hypotheses class - constitutes the template of our alignment function: a. Map each possible alignment into vectors in an abstract vector-space b. Devise a projection in the vector-space which order alignments in accordance to their quality 2. Derive a simple online learning algorithm 3. Convert the Online Alg. to a Batch procedure with some formal guarantees Phoneme Alignment. Slide 8

Feature “Primitives” for Alignment acoustic and phonetic representation feature primitive for alignment Assessing the quality of a suggested alignment Phoneme Alignment. Slide 9

Feature Primitive I Cumulative spectral change across the boundaries Phoneme Alignment. Slide 10

Feature Primitives I Cumulative spectral change across the boundaries Phoneme Alignment. Slide 11

Feature Primitives II Cumulative confidence in the phoneme sequence Learn a static frame-based phoneme classifier frame based phoneme classifier is the confidence that phoneme was uttered at frame (Dekel, Keshet, Singer, ‘ 04) Phoneme Alignment. Slide 12

Feature Primitive III Phoneme duration model - average length of phoneme - standard deviation of the length of phoneme Phoneme Alignment. Slide 13

Feature Primitive IV Speaking-rate (“dynamics”) Spectogram at different rates of articulation (Pickett, 1980) Phoneme Alignment. Slide 14

Feature Functions for Alignment Mapping all possible alignments into a vector space slightly incorrect alignment grossly incorrect alignment Phoneme Alignment. Slide 15

Main Solution Principle Find a linear projection that ranks alignments according to their quality slightly incorrect alignment grossly incorrect alignment Phoneme Alignment. Slide 16

Main Solution Principle (cont. ) example of low confidence projection slightly incorrect alignment grossly incorrect alignment Phoneme Alignment. Slide 17

Main Solution Principle (cont. ) example of incorrect projection slightly incorrect alignment grossly incorrect alignment Phoneme Alignment. Slide 18

Online Learning Algorithm Hypotheses class Cumulative cost Phoneme Alignment. Slide 19

Online Learning • For • Receive an instance • Predict • Receive true alignment and Pay cost • If • Set • Update Phoneme Alignment. Slide 20

Converting from Online to Batch • Run online algorithm on the training set and generate w 1, …, w. M • Small online error exists w 2 {w 1, …, w. M} whose generalization error is low (Cesa-bianchi et al. ) • Choose w 2 {w 1, …, w. M} which minimizes the error on a fresh validation set Phoneme Alignment. Slide 21

Algorithmic aspects • Running-time: • If the “inference”, , can be performed in polynomial time (e. g. dynamic programming), then the entire algorithm operates in polynomial time as well. • Worst case analysis for Online Learning: • For any competitor u, • Generalization error • Online-to-batch conversion guarantees that: low online error low generalization error Phoneme Alignment. Slide 22

Experiments • TIMIT corpus • Phoneme representation: • 48 phonemes (Lee & Hon, 1989) • Acoustic Representation: • MFCC+∆+∆∆ (ETSI standard) • TIMIT training set: • 500 utterances for training a frame classifier • 3096 utterances for learning alignment function • 100 utterances used for validation Phoneme Alignment. Slide 23

Alternative Approaches • Brugnara, Falavigna & Omologo, Automatic segmentation and labeling of speech based on HMM, 1993. • Hosom, Automatic phoneme alignment on acoustic-phonetic modeling, 2002. • Toledano, Gomez & Grande, Automatic Phoneme Alignment, 2003. Phoneme Alignment. Slide 24

Results Training size Test set Discrim. Algo. 650 or 3696 192 core 79. 7% 92. 1% 96. 2% 98. 1% Brugnara et al 3696 192 core 75. 3% 88. 9% 94. 4% 97. 1% Discrim. Algo. 650 or 2336 1344 entire 80. 0% 92. 3% 96. 4% 98. 2% Brugnara et al 2336 1344 entire 74. 6% 88. 8% 94. 1% 96. 8% t < 10 ms t < 20 ms t < 30 ms t < 40 ms Brugnara, Falavigna and Omologo, “Automatic segmentation and labling of speech based on Hidden Markov Models”, Speech Comm. , 12 (1993) 357 -370. Phoneme Alignment. Slide 25

Current and Future Work Discriminative learning methods for: • Whole phoneme sequence classification • 64% (ours) vs. 59% (HMM – IDIAP Torch 3) • Results without normalization of silences etc. • Small vocabulary continuous speech recognition • Segmentation of utterances to speakers • Full online learning setting: real-time adaptation to Speaker/environment changes Phoneme Alignment. Slide 26