Modeling Long Distance Dependence in Language Topic Mixtures

Modeling Long Distance Dependence in Language: Topic Mixtures Versus Dynamic Cache Models Rukmini. M Iyer, Mari Ostendorf 1

Introduction • Statistical language models – part of state-of-the-art speech recognizers • Word sequence: <s> w 1, w 2, …, w. T </s> • Consider the word sequence to be a Markov process • N-gram Probability: 2

Introduction • Use bigram representations – Bigram probability – Trigram in experiments • Although n-gram models are powerful, but are constrained to n. – Dependencies longer than n ? • The problem of representing long distance dependencies has been explored. 3

Introduction • Sentence-level: – words can co-occur in a sentence due to topic, grammatical dependencies • Article or conversation-level: – The subject of an article of a conversation • Dynamic cache language models address the second issue – Increase the likelihood of a word given that it has been observed previously in the article • Other methods: – Trigger language models – Context-free grammars 4

Mixture Model Framework • N-gram level: • Sentence level: Pk: bigram model for k-th class λk: mixture weight 5

Mixture Model Framework • Two main issues: – Automatic clustering to handle data not explicitly marked for topic dependence – Robust parameter estimation 6

Clustering • Specified by hand or determined automatically • Agglomerative clustering – Partition into desired number of topics • Performed at the article level – Reduce computation • The initial clusters can be singleton units of data 7

Clustering 1. let the desired number of clusters be C* and initial number of clusters be C 2. Find the best clusters, say Ai and Aj, to maximize some similarity criterion Sij 3. Merge Ai and Aj and decrement C 4. If current number of C=C*, then stop; otherwise go to Step 2 8

Clustering • Similarity measure: – Set-intersection measure – Combination of inverse document frequencies |Ai|: the number of unique words in cluster i |Aw|: the number of clusters containing the word w Nij: the normalization factor • Advantage of idf measure: – High-frequency words, such as function words are automatically discounted 9

Robust Parameter Estimation • Once the topic-dependent clusters are obtained, the parameters of the m components of the mixture can be estimated. • Two issues: – Initial clusters may not be optimal • EM • Viterbi-style – Sparse data problems • Double mixtures: – Sentence level – N-gram level 10

Iterative Topic Model Reestimation • Each component model in the sentence-level mixture is conventional n-gram model • Some back-off smoothing technique used – Witten-Bell technique • Clustering method is simple, articles may be clustered incorrectly, parameters may not be robust. – Iteratively reestimate 11

Iterative Topic Model Reestimation • Viterbi-style training technique: – Analogous to vector quantization design with LBG – Maximum likelihood is used as minimum distance – http: //www. data-compression. com/vq. shtml#lbg • Stop criterion: the size of the m clusters becomes nearly constant 12

Iterative Topic Model Reestimation • EM algorithm: – Incomplete data: do not know exactly which topic cluster a certain sentence belongs to – Theoretically better than Viterbi algorithm, although there is a little difference in practice – Potentially more robust – Complicated for language model training • Witten-bell back-off scheme 13

Iterative Topic Model Reestimation • E-step: expected likelihood of every sentence in the training corpus as belongs to each of the m topics. • A bigram model in the p-th iteration, the likelihood of the i-th training sentence as belong to the j-th topic is given: 14

Iterative Topic Model Reestimation • Derivation: 15

Iterative Topic Model Reestimation • M-step: • Bigram (wb, wc) probability for the j-th topic-model : 16

Iterative Topic Model Reestimation • Unigram back-off: 17

Iterative Topic Model Reestimation • Meaning: Witten-Bell 18

Iterative Topic Model Reestimation • MLE for bigram (wb, wc): • Unigram wc: 19

Topic Model Smoothing • Smoothing due to sparse data: – Interpolate with general model at n-gram level • Come across nontopic sentences: – Including a general model in addition to the m component models at sentence level 20

Topic Model Smoothing • Weight are estimated separately using a held-out data 21

Dynamic adaptation • The parameters (λk, P(wi|wi-1)) of the sentencelevel mixture model are not updated as we observe new sentence. • Dynamic language modeling tries to capture short-term fluctuations in word frequencies. – Cache or trigger model (word frequency) – Dynamic mixture weight adaptation (topic) 22

Dynamic Cache Adaptation • The cache maintains a list of counts of recently observed n-grams • Static language model probabilities are interpolated with the cache n-gram probabilities. • Specific topic-related fluctuations can be captured by using a dynamic cache model if the sublanguage reflects style. 23

Dynamic Cache Adaptation • Two important issues to dynamic language modeling with a cache: – The definition of cache – The mechanism for choosing the interpolated weight • The probabilities of content words vary with time within the global topic • Small but consistent improvements over the frequency-based rare-word cache 24

Dynamic Cache Adaptation • The equation for adapted mixture model incorporating component-caches: • μ: empirically estimated on a development test set to minimize WER 25

Dynamic Cache Adaptation • Two approaches for updating the word counts – Partially observed documents is cached and the cache is flushed between documents. – A sliding window is used to determine the contents of the cache if the document or article is reasonably long. • Select first approach – Do not have long articles 26

Dynamic Cache Adaptation • Extend cache-based n-gram adaptation to the sentence -level mixture model • Fractional counts of words are assigned to each topic according to their relative likelihoods • The likelihood of the k-th model given the i-th sentence yi with words w 1, …, w. T: 27

Dynamic Cache Adaptation • Derivation: 28

Dynamic Cache Adaptation • Meaning: zi 1 zi 2 Sentence i n features k topic caches zik 29

Mixture Weight Adaptation • The sentence-level mixture weights are updated with each new observed sentence, to reflect the increased information on which to base the choice of weight. 30

Experiments • Corpus: – NAB news, not marked for topics – Switchboard, 70 topics • Recognition system: – Boston University system • Non-parametric trajectory stochastic segment model – BBN byblos system: • Speaker-independent HMM system 31

Experiments 32

Experiments 33

Experiments 34

Conclusions • Investigate a new approach to language modeling using a simple variation of the n-gram approach: sentence-level mixture • Automatic clustering algorithm to classify text as one of m different topics • Other language modeling advances can be easily incorporated in this framework. 35