Test pattern speech analysis Filter bank LPCC MFCC

통계적 패턴매칭에 의한 음성인식 방법 Test pattern speech analysis Filter bank LPCC MFCC Pattern training Templates or models Reference pattern Pattern classifier Recognized speech Decision logic Dynamic time warping Search algorithm 2

마코프 프로세스의 예 날씨 추정 문제 – State 1: rain or snow – State 2: cloudy – State 3: sunny 0. 4 0. 3 Rain 1 or snow 문제: Probability of “sunsun-rain-suncloudy-sun”? 해결책: P(O|M)=P[3, 3, 3, 1, 1, 3, 2, 3|M] =P[3]P[3|3]2 P[1|3]P[1|1]P[3|1 ]P[2|3]P[3|2] =1. 536 x 10 -4 0. 6 2 Cloudy 0. 2 0. 1 0. 3 0. 2 3 Sunny 0. 8 6

히든 마코프 모델 (Hidden Markov Model: HMM) 이론 이중 랜덤 프로세스 b 1(x) – Hidden probabilistic state transition process – Observable probabilistic output process 상태 천이확률(State transition probability), aij=p(sj|si) 출력 확률분포(Output probability distribution), bi(x)=p(x|si) aii A B C D aij ajj b 2(x) A B C D b 3(x) A B C D State seq. : s 1 s 3 s 1 s 2 s 3 Observed seq. (X): A C D B A C 7

HMM을 정의하는 파라메터들 파라메터 의미 예제 N Number of states Number of urns M Number of distinct observation symbols Number of distinct colors A State-transition probability distribution Random procedure to select a new urn B Observation symbol probability distribution Color distribution of each urn p Initial state distribution Probability of selecting an urn initially 9

Forward Algorithm 관측 이벤트열 X와 모델 파라메터 set l가 주어졌을 때, 확률 P(X|l)를 효율적으로 계산하는 방법. Direct computation – 계산량: TNT Forward algorithm – – – Define Initialization Recursion Termination 계산량 : N 2 T j i 12

Baum-Welch Algorithm (II) 16

HMM 구조의 종류 Ergodic model – Every aij is positive. – Every transition is possible. Left-to-right (Bakis) model – aij = 0, j<i – Cannot go backward – Good to model temporal structure of speech signals 17

HMM 출력확률분포의 종류 Discrete HMM: – Discrete symbols Continuous HMM: one codebook per state bj(k) – Mixture Gaussian pdf bj(x) Semi-continuous HMM: globally shared codebook bj(x) codebook 18

연속밀도(Continuous) HMM Re-estimation formula 19

음성인식에 유용한 HMM의 변형 Null Transition – Produces no output – Jumps from one state to another that produce no observation – Makes network simple f f Explicit State Duration Modeling – State duration of a state with selftransition probability aii has exponential distribution: pi(d)=aiid-1(1 -aii) – Actual state duration has Gamma distribution 20

HMM 파라메터 추정 방법들 Maximum likelihood estimation (Baum. Welch algorithm based on EM algorithm) l’ = argmaxl P(X|l) Maximum mutual information estimation l’ = argmaxl Il(W; X), W: word Discriminative training for minimum classification error – Minimum recognition error training – Generalized probabilistic descent – Corrective training 21

HMM을 이용한 음성인식기 구현 시 실제 고려사항 (I) Scaling – To prevent underflow – Use normalized a – Same recursion Multiple observation – Re-estimation with L observations 23

HMM을 이용한 음성인식기 구현 시 실제 고려사항 (II) Initial estimates of HMM parameters – HMM converges to local maximum and therefore good initial parameter estimates are necessary for rapid and proper convergence. – p, A: random or uniform – B: Essential • Manual segmentation of observation sequence into states and get average of observations within states • Maximum likelihood segmentation of observations and averaging • Segmental k-means segmentation with clustering 24

HMM을 이용한 음성인식기 구현 시 실제 고려사항 (III) Size of Training Data – Small data may cause under-estimation problem – Solution: • Increase training data: The more, the better • Reduce the model size or parameter tying (sharing) • Enhance reliability of parameter estimates: Deleted interpolation Choice of Model – – Task-dependency Number of states HMM type: Ergodic or left-to-right Observation densities: Discrete or continuous, single or mixture Gaussian 25

HMM과 음성신호의 대응관계 Assume speech signals are generated by HMM Find model parameters from training data Compute probability of test speech using the HMM and select the model having maximum likelihood Left-to-right model: easy to model signal whose properties change over time in a successive manner – e. g. , speech HMM ㅕ ㄹ ㅓ Features frame Frame shift Speech signals time 26

HMM에 의한 단어모델 훈련과정 waveform feature il i chil Speech database Feature Extraction Baum-Welch Re-estimation Converged? Yes end No l 1 Word HMM l 2 l 7 27

훈련 전에 미리 정의해야 하는 모델 파라메터들 Number of states (N) – Number of phones in a word • Digit recognition: English N=9, Korean N=6 – Average number of observations in a spoken word Feature – LPC-cepstrum or MFCC – Derivatives Observation density – Discrete: M=512 -1024 codewords – Continuous: M=1 -64, Diagonal covariance 28

단어단위 HMM에 의한 고립단어 인식과정 l 1 Seoul HMM for word 1 Likelihood computation Speech Feature extraction P(X|l 1) . . . l. V HMM for word V Likelihood computation Select maximum Recognized word P(X|l. V) 29

단어단위 HMM 인식 성능평가 예 Training data: 10 digit x 100 speakers Test data: 10 digit x 100 speakers Results (Error rates): – Speaker-dependent test • LPC/DTW 1. 6% • CD/HMM 1. 6% LPC/VQ/DTW (discrete) 3. 5% VQ/HMM (discrete) 3. 7% – Speaker-independent test • LPC/DTW 1. 6% • CD/HMM 1. 6% Discussion: – When using VQ, performance degrades. – DTW and CHMM are comparable Ref: Rabiner & Juang, Fundamentals of Speech Recognition 30

음소모델 훈련 및 고립단어 인식 개요 Test word 일 Feature extraction Likelihood computation Recognized word Decision rule i Word model generation l i s a HMM “일” HMM “이” m HMM “삼” “i” “s” “l” “a” … Training words subword model lexicon 일il 이i 삼sam Subword model generation 31

음소모델 훈련과정 waveform feature il i chil Speech database Feature Extraction Baum-Welch Re-estimation Word model generation Yes end Converged? No li lexicon subword model ll lch 32

Segmental K-means 훈련방법 Labeled utterance/ Unlabeled Transcriptions feature i l i ch i l Feature Extraction Speech database i Viterbi segmentation ch K-means clustering, accumulate statistics l i Build word model i ch l il i chil i l Phone model i l ch lexicon Model update ii il i l chil ch i l No Termination condition Yes end 33

음향모델 단위 비교 Word unit – – A word consists of an HMM High accuracy for small vocabulary A large number of parameters Should retrain models if vocabulary changes Word: 일 il 이i 삼 sam 칠 chil Subword unit – A word HMM consists of combination of smaller units than word – Can build a new word model by combining subword models – Can build vocabulary-independent systems – Needs small training data because units are shared among words Phoneme: 일il 이i 삼sam 칠 ch i l 34

연속음성인식의 학제간 연구 필요성 음성학: relationship between speech signal and human vocal tract & hearing mechanisms 언어학: phonology, syntax, semantics 통신이론 및 정보이론: parameter estimation, coding and decoding algorithms (stack decoding, Viterbi decoding), Information-theoretic distance measures 신호처리: spectral analysis, feature extraction, time-varying signal modeling, noise-robust features 패턴인식: cluster speech patterns and match, Bayesian learning 전산학: efficient search algorithms, etc 36

HMM 기반 연속음성인식 과정 Speech signals Word sequence Feature extraction Search one One two three. Network construction two oh Speech database Text corpora HMM estimation LM estimation Acoustic model Vocabulary Language Dictionary model one two nine oh 38

대표적인 특징추출 방법 Hamming window Mel-scale filter bank FFT log |. | DCT weight x(n) 1 0 1 2 MFCC Filters 3 4 freq(k. Hz) 39

음향모델링 Subword Units – Context-independent • Phoneme • Syllable • Small number of parameters – Context-dependent • • • Diphone Triphone Quinphone High accuracy Large number of units Large training data Diphone: 일 sil-i i-l l-sil 이 sil-I I-sil 삼 sil-s s-a a-m m-sil Triphone: 일 sil-i+l i-l+sil 이 sil-i+sil 삼 sil-s+a s-a+m a-m+sil 40

음향모델 훈련용 네트워크 구성 Sentence model = (word 1 word 2 … word. N) Word model = (phone 1 phone 2 … phone. M) Phone model = (state 1 state 2 state 3) ONE Sentence HMM TWO THREE AH N ONE TWO THREE ONE W Word HMM ONE Phone HMM W 1 2 3 41

강인한 음향모델링을 위한 파라메터 공유 (I) Share parameters (mean, variance, transition probabilities) if two states have similar distributions – Data-driven clustering t-ih+ng f-ih+l s-ih+l (continued) 42

강인한 음향모델링을 위한 파라메터 공유 (II) – Decision tree-based clustering • For large vocabulary system • Can handle unseen triphones • How can we get model for an unseen triphone t-aw+ih? s-aw+n Cluster center states of phone /aw/ t-aw+n s-aw+t R=consonant? n L=nasal? n y y R=nasal? y L=stop? n y n 2 3 4 5 1 States in each leaf node are tied 43

인식용 문법 네트워크 구성 one call John Tom two one START nine START two oh nine One two one three Seven oh two five … oh Call John Call Tom One two one three Seven oh two five … 45

인식용 전체 네트워크 상세구조 I start Intra-word transition Word transition end 이 P(이 |x) LM is applied P(일 |x) I L 일 S A P(사 |x) Between-word transition 사 S A M P(삼 |x) 삼 46

모델간 연결 규칙 Intra-word transitions Between-word transitions – Add LM score and word insertion penalty Acoustic loglikelihood LM score Word insertion penalty 47

연속음성인식을 위한 검색 알고리즘 Search is a problem for finding the optimal path from search network. One-Pass (One-Stage) dynamic programming algorithm – Extension of dynamic programming (DP) strategies for connected pattern matching – Fast, Simple, Efficient, Small memory – Frame synchronous – Widely used for continuous speech recognition 48

DTW와 Viterbi 알고리즘 비교 DTW v. s. Viterbi algorithm – Reference template State – Path constraint Transition prob – Local distance Observation prob Reference state 5 4 3 2 s 2 2 1 s 3 5 1 0 0 1 2 3 4 5 6 Frame (time) 49

One-Pass DP 알고리즘 Representation in lattice Word State index 공 V 구 k S(k) (t, i, k) i 1 Between-word transition 이 2 Intra-word transition S(1) 일 1 1 1 t T 50

연속음성인식의 성능평가 방법 Errors in continuous speech recognition – – Insertion (I) Deletion (D) Substitution (S) Number of test words (N) Word error rate (E)=(I+D+S)/N Word correct (C)=1 -(D+S)/N Word accuracy=1 -E REF: 일 이 칠 구 오 일 삼 HYP: 일 칠 구 오 이 일 사 After aligning REF: 일 이 칠 구 오 * 일 삼 HYP: 일 * 칠 구 오 이 일 사 Results S=1 I=1 D=1 E=3 N=7 Correct=71% Accuracy=57% 52

연속숫자음 인식 예 (I) Feature extraction – – – – Sampling rate = 6. 67 k. Hz Window size = 300 samples (45 ms) Window shift = 100 samples (15 ms) LPC order = 8 Cepstrum order = 12 Delta and delta-delta cepstrum = 12 and 12 Cepstral window: raised sine-like window Acoustic modeling – – N-state word model (N varied from 5 to 10 for different digits) Continuous mixture density, M=64 Log-energy probability density within state State duration density Post-processing – A single Gaussian digit duration density 53

연속숫자음 인식 예 (II) TIDIGITS – – Training: 77 strings x 112 speakers (length 1 -5 or 7 digits) Test: 77 strings x 113 speakers Low-pass filtered to telephone bandwidth String error rates (%) of speaker independent test results • Unknown length (UL) • Known length (KL) Mode SI Training data Testing data UL KL 0. 3 0. 05 1. 4 0. 8 f start Grammar Network f 0 sil 1 digit f 2 end 3 sil 54

결 론 (I) Statistical approach to speech recognition is successful. Double random processes are in HMM: state transition and observation probabilities. Three basic problems for HMM to be applied in realworld applications have been solved. HMM is good for modeling temporal structure of speech signals. Reviewed HMM types and observation probability distributions for speech recognition. HMM-based speech recognizers have great flexibility and good accuracy. 55

결 론 (II) Advantages of HMM-based CSR – Simple algorithmic structure – Flexibility: Easily changeable and extendible to another vocabulary – Solid theoretical background – Usefulness has been proved by real products Disadvantages – Requires large training data – Weak discrimination power (ML estimation) – Does not match real speech characteristics 56

참고문헌 X. Huang, et al. , Spoken Language Processing, Prentice Hall, 2001. L. R. Rabiner and B. -H. Juang, Fundamentals of Speech Recognition, Prentice Hall, 1993. H. Ney, “The use of a one-stage dynamic programming algorithm for connected word recognition, “ IEEE Trans. ASSP, vol. 32, no. 2, pp. 263 -271, Apr. 1984. J. -C. Junqua and R. Kuhn, “Adaptive Methods for Speech and Speaker Recognition, ” ICSLP 2002 Tutorial, 2002. S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland, The HTK Book Version 3. 0, Microsoft, 2000. B. I. Pawate and P. D. Robinson, “Implementation of an HMMBased speaker-independent speech recognition system on the TMS 320 C 2 x and TMS 320 C 5 x, ” Texas Instruments Inc. , Application Report. 1996. 57