A Tutorial of HMM Tool Kit HTK Hongbing

A Tutorial of HMM Tool Kit (HTK) Hongbing Hu Hidden Markov Models Electrical and Computer Engineering Binghamton University Speech Recognition Department of Electrical and Computer Engineering Binghamton University, State University of New York 12/02/2008 1

Outline The Fundamentals of Speech Recognition q An Overview of HTK q A Tutorial Example of Using HTK q Further Topics on HTK q Department of Electrical and Computer Engineering Binghamton University, State University of New York 2

The Fundamentals of Speech Recognition Architecture q Bayes Rule q Hidden Markov Models (HMMs) q Gaussian Mixture Models q Viterbi Decoding q Continuous Speech Recognition q Department of Electrical and Computer Engineering Binghamton University, State University of New York 3

Speech Recognition Architecture Speech Waveform Feature Extraction Speech Features Classification (Recognition) Words ai n iy d sil e Phonemes Recognizer (HMM/NN) I need a Department of Electrical and Computer Engineering Binghamton University, State University of New York 4

Bayes Rule Likelihood Prior Posterior Evidence Prior: Recognition unit (word, phoneme …) probability determined by language modeling q Evidence: A scale factor determined by prior and likelihood to guarantee that the posteriors sum to 1 q Likelihood: Likelihood between a sequence of speech vectors (O) and a candidate unit (wi) , could be estimated by HMMs q Department of Electrical and Computer Engineering Binghamton University, State University of New York 5

Hidden Markov Models (HMMs) Speech vectors of a unit are generated by a Markov model q The overall probability is calculated as the product of the transition and output probabilities q q Likelihood can be approximated by only considering the most likely state sequence Department of Electrical and Computer Engineering Binghamton University, State University of New York 6

Gaussian Mixture Models q Gaussian Mixture Models Output distributions are represented by Gaussian Mixture Models q Each observation vector is allowed to be split into a number of independent data streams (S), consisting of multiple mixtures (Ms) q q Multivariate Gaussian Department of Electrical and Computer Engineering Binghamton University, State University of New York 7

Viterbi Decoding q Find the best path through a matrix representing the states of the HMM and the frames of speech Department of Electrical and Computer Engineering Binghamton University, State University of New York 8

Continuous Speech Recognition q Difficulties Training data must consist of continuous utterances q The boundaries between units will not be known q Label Training Data by Hand q Embedded Training q q Construct a composite HMM through joining the HMMs corresponding to the units of the training utterance Department of Electrical and Computer Engineering Binghamton University, State University of New York 9

An Overview of HTK q HTK: A toolkit for building Hidden Markov Models HTK is primarily used for speech recognition research q HTK has also been used for numerous other applications including research into speech synthesis, character recognition and DNA sequencing q HTK was originally developed at the Machine Intelligence Laboratory of the Cambridge University Engineering Department (CUED) q Microsoft acquired the license of HTK in 1999, but is still providing it as open source q Department of Electrical and Computer Engineering Binghamton University, State University of New York 10

Features of HTK World recognized state-of-the-art speech recognition system q HTK can support a variety of different formats q q q ex : pcm, wav, …, ALIEN(unknown), etc. Feature extraction: q MFCC, filterbank, PLP, LPC, …, etc. Very flexible HMM definition q Training q Viterbi(segmentation) q Forward/Backward (Baun-Welch) q Department of Electrical and Computer Engineering Binghamton University, State University of New York 11

Software Architecture Much of the functionality is built into the library modules q HTK tools are designed to run with a traditional command line style interface q q Configuration files control detailed behaviors of tools Department of Electrical and Computer Engineering Binghamton University, State University of New York 12

HTK Processing Stages q Data Preparation q Training q Testing/Recognition q Analysis Department of Electrical and Computer Engineering Binghamton University, State University of New York 13

Data Preparation Phase q A set of speech data files and their associated transcriptions are required Convert the speech data files into an appropriate acoustic feature format (MFCC, LPC) [HCopy, HList] q Convert the associated transcriptions into an appropriate format which consists of the required phone or word labels [HLEd] q Department of Electrical and Computer Engineering Binghamton University, State University of New York 14

HSLAB q Record the speech and manually annotate it with any required transcriptions if the speech needs to be recorded Department of Electrical and Computer Engineering Binghamton University, State University of New York 15

HCopy q Parameterize the speech waveforms to a variety of acoustic feature formats by setting the appropriate configuration variables Department of Electrical and Computer Engineering Binghamton University, State University of New York MFCC: Mel-Frequency Cepstral Coefficients LPC: Linear Prediction filter Coefficients LPCEPSTRA: LPC cepstral coefficients DISCRETE: Vector quantized data 16

Other Data Preparation Tools q HList q q HLEd q q A script-driven text editor used to make the required transformations to label files, for example, the generation of context-dependent label files HLStats q q Check the contents of any speech file and the results of any conversions before processing large quantities of speech data Gather and display statistical information on the label files HQuant q Build a VQ codebook in preparation for build discrete probability HMM systems Department of Electrical and Computer Engineering Binghamton University, State University of New York 17

Training Phase q Prototype HMMs Define the topology required for each HMM by writing a prototype Definition q HTK allows HMMs to be built with any desired topology q HMM definitions stored as simple text files q All of the HMM parameters (means and variances) are ignored except the transition probabilities q Department of Electrical and Computer Engineering Binghamton University, State University of New York 18

Training Procedure Isolated Word Training [Hinit, HRest] Flat Start Training [HComp. V] Embedding Training [HERest] Department of Electrical and Computer Engineering Binghamton University, State University of New York 19

Isolated Word Training q HInit Read in all of the bootstrap training data and cuts out all of the examples of a specific phone q Iteratively compute an initial set of parameter value using the segmental k-means training procedure q On the first iteration cycle, the training data are uniformly segmented with respective to its model state sequence, and then means and variances are estimated q On the successive iterations cycles, the uniform segmentation is replaced by Viterbi alignment q q HRest q q Re-estimate the parameters initially computed by HInit Baum-Welch re-estimation procedure is used, instead of the segmental k-means training procedure for HInit Department of Electrical and Computer Engineering Binghamton University, State University of New York 20

Flat Start Training If the training speech files are not equipped the sub-word -level boundary information, a flat-start training scheme can be used q HComp. V q q All of the phone models are initialized to be identical and have state means and variances equal to the global speech mean and variance Department of Electrical and Computer Engineering Binghamton University, State University of New York 21

Embedding Training q HERest Perform a single Baum. Welch re-estimation of the whole set of the HMMs simultaneously q The corresponding phone models are concatenated and the forward/backward algorithm is used to accumulate the statistics of state occupation, means, variances, etc. , q The accumulated statistics are used to re-estimate the HMM parameters q Department of Electrical and Computer Engineering Binghamton University, State University of New York 22

Recognition Phase q HVite q Perform Viterbi-based speech recognition Dictionary: define how each word is pronounced q Word networks: either simple word loops where any word can follow any other word or directed graphs representing a finitestate task grammar - HBuild and HParse are supplied to create the word networks q Support cross-word triphones and run with multiple tokens to generate lattices containing multiple hypotheses q Configured to rescore lattices and perform forced alignments q Department of Electrical and Computer Engineering Binghamton University, State University of New York 23

Analysis Phase q Evaluate the recognizer performance by comparing the recognition results q HResults q Perform the comparison of recognition results and correct reference transcriptions by using dynamic programming to align them Department of Electrical and Computer Engineering Binghamton University, State University of New York 24

HResults q Provide an alternative similar to that used in the US NIST scoring package q Provide confusion matrices, speaker-by-speaker breakdowns and time-aligned transcriptions Department of Electrical and Computer Engineering Binghamton University, State University of New York 25

A Tutorial Example q A voice-operated interface for phone dialing Dial three two six five four Dial nine zero four one oh nine Phone Woodland Call Steve young All of the speech is recorded from scratch q A task grammar and a pronunciation dictionary are defined q Department of Electrical and Computer Engineering Binghamton University, State University of New York 26

Processing Steps q Data Preparation q q q Training HMMs q q Step 1 – the Task Grammar Step 2 – the Dictionary Step 3 – Recording the Data Step 4 – Creating the Transcription Files Step 5 – Coding the Data Step 6 – Creating Flat Start Monophones Recognition Step 7 – Recognizing the Test Data q Step 8 – Evaluating the Performance q Department of Electrical and Computer Engineering Binghamton University, State University of New York 27

Step 1 – the Task Grammar q Specify simple task grammars with the HTK grammar definition language gram This high level representation is provided for user convenience Department of Electrical and Computer Engineering Binghamton University, State University of New York 28

Step 1 – the Task Grammar (cont. ) q A low level notation (HTK Standard Lattice Format ) is required in the actual HTK recognizer q Standard Lattice Format (SLF) q Each word instance and each word-to-word transition is listed explicitly SLF example Department of Electrical and Computer Engineering Binghamton University, State University of New York 29

Step 2 – the Dictionary dict Function words such as A and TO have multiple pronunciations q The entries (SENTSTART, SENTEND) have a silence model sil as their pronunciations and null output symbols q Department of Electrical and Computer Engineering Binghamton University, State University of New York 30

Step 3 – Recording the Data q Generate utterances testprompts q Record the speech Department of Electrical and Computer Engineering Binghamton University, State University of New York 31

Step 4 – Creating the Transcription Files Phone Level Transcription Department of Electrical and Computer Engineering Binghamton University, State University of New York 32

Step 5 – Coding the Data Coding can be performed using the tool HCopy configured to automatically convert its input into feature (MFCC) vectors q Configuration file (config) q Department of Electrical and Computer Engineering Binghamton University, State University of New York 33

Step 5 – Coding the Data (cont. ) codetr. scp Department of Electrical and Computer Engineering Binghamton University, State University of New York 34

Step 6 – Creating Flat Start Monophones q Define Prototype HMMs 3 -state left-right with no skips q Proto HMMs files are stored individually (hmm 0) q q Master Macro File (MMF) Department of Electrical and Computer Engineering Binghamton University, State University of New York ~o <Vec. Size> 39 <MFCC_0> ~h "w" <Begin. HMM> ~o <Vec. Size> 39 <MFCC_0> <Num. States> 5 ~h “n" <State> 2 <Begin. HMM> <Mean>3939<MFCC_0> ~o <Vec. Size> <Num. States> 5 ~h “ah"0. 0. . . <State> 2 <Variance> 39 <Begin. HMM> <Mean> 39 1. 0 <Num. States> 5. . . 0. 0. . . …… <State> 2 <Variance> 39 <State> <Mean> 39 4 1. 0. 0<Mean> 0. 0. . . 39 …… 0. 0390. 0. . . <Variance> <State> 4 39 1. 0<Variance> 1. 0. . . <Mean> 39 …… 1. 0. 0 0. 0. . . <Trans. P> 5 <State> 4 <Variance> 39 0. 039 1. 0 0. 0 <Mean> 1. 0. 00. 00. 6. . . 0. 4 0. 0 <Trans. P> 5 0. 0390. 6 0. 4 0. 0 <Variance> 0. 0 1. 0 0. 0 1. 00. 0. 0 0. 7 0. 3 0. 0 0. 6 0. 4 0. 0 <Trans. P> 5 0. 0 0. 6 0. 4 0. 0 1. 0 0. 0 <End. HMM> 0. 0 0. 7 0. 3 0. 0 0. 6 0. 4 0. 0 0. 6 0. 4 0. 0 <End. HMM> 0. 0 0. 7 0. 3 0. 0 0. 0 <End. HMM> 35

Step 6 – Creating Monophones (cont. ) Department of Electrical and Computer Engineering Binghamton University, State University of New York 36

Step 7 – Recognizing the Test Data q Viterbi-based Speech Recognition (HVite) HVite -H hmm 0/macros -H hmm 0/hmmdefs -S test. scp -l ’*’ -i recout. mlf -w wdnet -p 0. 0 -s 5. 0 dict monphones 0 Department of Electrical and Computer Engineering Binghamton University, State University of New York 37

Step 8 – Evaluating the Performance HResults -I testref. mlf monphones 0 recount. mlf SENT: sentence level statistics q WORD: word level statistics q Department of Electrical and Computer Engineering Binghamton University, State University of New York 38

Further Topics on HTK q HMMs Training with labeled speech databases The TIMIT Database: A total of 6300 sentences(630 speakers) from 8 major dialect regions of the U. S. q Speech data and labeled Transcriptions are provided q q Tied-State Triphone HMMs Department of Electrical and Computer Engineering Binghamton University, State University of New York 39

HMMs Training with TIMIT q Data Preparation q q q Training HMMs q q Step 1 – the Grammar Speech data and labeled Transcriptions are provided Step 2 – the Dictionary Step 3 – Recording the Data Step 4 – Creating the Transcription Files Step 5 – Coding the Data Step 6 – Creating Monophones Recognition Step 7 – Recognizing the Test Data q Step 8 – Evaluating the Performance q Department of Electrical and Computer Engineering Binghamton University, State University of New York 40

Step 1 – the Grammar q HLStats q A bigram language model can be built HLStats –b stats –o wordlist train. mlf q HBuild data ngram 1=50 ngram 2=1694 1 -grams: -99. 999 -1. 8048 -1. 7979 …… 2 -grams: -0. 0001 -2. 8093 …… !ENTER aa ae stats -3. 8430 -1. 4168 -1. 6101 !ENTER si aa aa aa ao construct word-loop and word-pair grammars automatically q It can incorporate a statistical bigram language model into a network. q HBuild –n stats wdnet train. mlf Department of Electrical and Computer Engineering Binghamton University, State University of New York 41

Step 2 – the Dictionary q Phone Level Dictionary Create phone level dictionary for phonetic recognition q The recognition output is phonemes instead of words q sil iy iy vv ih ih nn eh eh ix ix ff dict …… yy en en zh zh !ENTER [] sil !EXIT [] sil Department of Electrical and Computer Engineering Binghamton University, State University of New York 42

Step 6 – Creating Monophones Isolated Word Training q Initialization with Hinit q HInit –S trainlist –I train. mlf –M hmm 0 proto (sil, . . . ) q Re-estimate the parameters with HRest –S trainlist –I train. mlf -u tmvw -w 3 -i 20 –M hmm 1 hmm 0 (sil, . . . ) Department of Electrical and Computer Engineering Binghamton University, State University of New York 43

Triphone HMMs q Context-dependent Triphone HMMs q A powerful subword modeling technique because they account for the left and right phonetic contexts Text: men need … Monphone: sil m ae n sp n iy d. . . Triphone: sil m+ae m-ae+n ae-n sp n+iy n-iy+d iy-d … Department of Electrical and Computer Engineering Binghamton University, State University of New York 44

q Convert the monophone transcriptions to triphone transcriptions mktri. led q Triphones can be made by simply cloning monophones and re-estimating using triphone transcriptions mktri. hed Department of Electrical and Computer Engineering Binghamton University, State University of New York 45

Tied-State Triphone HMMs q Triphone Clustering Too many models for the amount of training data available, Some models do not have sufficient data for training q Many triphone contexts are very similar q q Decision Tree State Tying Department of Electrical and Computer Engineering Binghamton University, State University of New York 46

References q “The HTK Book, ” Steve Young, et al. , Dec. 2006 q q “Introduction to HTK Toolkit, ” Berlin Chen, 2004 q q http: //htk. eng. cam. ac. uk/docs. shtml http: //140. 122. 185. 120/Past. Courses/2004 -TCFSTAudio%20 and%20 Speech%20 Recognition/Slides/SP 2004 F_Lect ure 06_Acoustic%20 Modeling-HTK%20 Tutorial. pdf HTK Download q http: //htk. eng. cam. ac. uk/download. shtml (User registration required) Department of Electrical and Computer Engineering Binghamton University, State University of New York 47