Information Extraction using HMMs Sunita Sarawagi 1 IE

Information Extraction using HMMs Sunita Sarawagi 1

IE by text segmentation Source: concatenation of structured elements with limited reordering and some missing fields n Example: Addresses, bib records House number Building Road City State Zip 4089 Whispering Pines Nobel Drive San Diego CA 92122 Author Year Title Journal Volume Page P. P. Wangikar, T. P. Graycar, D. A. Estell, D. S. Clark, J. S. Dordick (1993) Protein and Solvent Engineering of Subtilising BPN' in Nearly Anhydrous Organic Media J. Amer. Chem. Soc. 115, 12231 -12237. 2

Hidden Markov Models n Doubly stochastic models A 0. 6 A 0. 9 C 0. 5 0. 1 S 1 Efficient dynamic programming algorithms exist for n n n Finding Pr(S) The highest probability path P that maximizes Pr(S, P) (Viterbi) 0. 4 S 2 0. 5 0. 1 n 0. 9 C 0. 8 S 3 S 4 A 0. 5 0. 2 A 0. 3 C C 0. 5 0. 7 Training the model n (Baum-Welch algorithm) 3

Input features § Content of the element § Specific keywords like street, zip, vol, pp, § Properties of words like capitalization, parts of speech, number? § § Inter-element sequencing Intra-element sequencing Element length External database § Dictionary words § Semantic relationship between words § Frequency constraints 4

IE with Hidden Markov Models n Probabilistic models for IE Emission probabilities Transition probabilities 0. 5 Author Y 0. 1 A 0. 1 C 0. 8 Year dddd 0. 8 dd 0. 2 0. 9 Title 0. 5 A 0. 6 B 0. 3 C 0. 1 0. 8 Journal 0. 2 X 0. 4 B 0. 2 Z 0. 4 5

HMM Structure n Naïve Model: One state per element Nested model Each element another HMM n 6

Comparing nested models n Naïve: Single state per tag n n n Element length distribution: a, a 2, a 3, … Intra-tag sequencing not captured Chain: n Element length distribution: n n n Each length gets its own parameter Intra-tag sequencing captured Arbitrary mixing of dictionary, n n n Eg. “California York” Pr(W|L) not modeled well. Parallel path: n n Element length distribution: each length gets a parameter Separates vocabulary of different length elements, (limited bigram model) 7

Embedding a HMM in a state 8

Bigram model of Bikel et al. n Each inner model a detailed bigram model n n First word: conditioned on state and previous state Subsequent words conditioned on previous word and state Special “start” and “end” symbols that can be thought Large number of parameters n n (Training data order~60, 000 words in the smallest experiment) Backing off mechanism to previous simpler “parent” models (lambda parameters to control mixing) 9

Separate HMM per tag n Special prefix and suffix states to capture the start and end of a tag Road name Prefix S 1 S 2 S 4 Suffix Building name Prefix S 1 S 2 S 3 S 4 Suffix 10

HMM Dictionary n For each word (=feature), associate the probability of emitting that word n n Multinomial model Features of a word, n example, n n n part of speech, capitalized or not type: number, letter, word etc Maximum entropy models (Mc. Callum 2000), other exponential models Bikel: <word, feature> pairs 11

Feature Hierarchy 12

Learning model parameters n When training data defines unique path through HMM n Transition probabilities n Probability of transitioning from state i to state j = number of transitions from i to j total transitions from state i n Emission probabilities n Probability of emitting symbol k from state i = number of times k generated from i number of transition from I n When training data defines multiple path: n A more general EM like algorithm (Baum-Welch) 13

Smoothing n Two kinds of missing symbols: n n n Case-1: Unknown over the entire dictionary Case-2: Zero count in some state Approaches: n Laplace smoothing: ki + 1 m + |T| n Absolute discounting n n P(unknown) proportional to number of distinct tokens P(unknown) = (k’) x (number of distinct symbols) P(known) = (actual probability)-(k’), k’ is a small fixed constant, case 2 smaller than case 1 14

Smoothing (Cont. ) n Smoothing parameters derived from data n n Partition training data into two parts Train on part-1 Use part-2 to map all new tokens to UNK and treat it as new word in vocabulary OK for case-1, not good for case-2. Bikel et al use this method for case-1. For case-2 zero counts are backed off to 1/(Vocab-size) 15

Using the HMM to segment n n Find highest probability path through the HMM. Viterbi: quadratic dynamic programming algorithm 115 Grant street Mumbai 400070 115 Grant ………. . 400070 House Road City Pint o o Pint 16

Most Likely Path for a Given Sequence n The probability that the path sequence is generated: transition probabilities is taken and the emission probabilities 17

Example 0. 4 0. 2 0. 5 A C G T 0. 4 0. 1 0. 2 0. 3 0. 8 A C G T 1 begin 0 0. 5 A C G T 0. 4 0. 1 0. 4 0. 6 3 0. 2 2 0. 8 0. 2 0. 3 0. 2 A C G T 0. 1 0. 4 0. 1 end 5 0. 9 4 0. 1 18

Finding the most probable path: the Viterbi algorithm n n define to be the probability of the most probable path accounting for the first i characters of x and ending in state k we want to compute , the probability of the most probable path accounting for all of the sequence and ending in the end state can define recursively can use dynamic programming to find efficiently 19

Finding the most probable path: the Viterbi algorithm n initialization: 20

The Viterbi algorithm n recursion for emitting states (i =1…L): keep track of most probable path 21

The Viterbi algorithm n n termination: to recover the most probable path, follow pointers back starting at 22

Database Integration n Augment dictionary n n n Example: list of Cities Assigning probabilities is a problem Exploit functional dependencies n Example n n Santa Barbara -> USA Piskinov -> Georgia 23

2001 University Avenue, Kendall Sq. Piskinov, Georgia House number Road Name 2001 House number Area City State University Avenue, Kendall Sq. , Piskinov, Georgia Road Name Area City Country 2001 University Avenue, Kendall Sq. , Piskinov, Georgia 24

Frequency constraints n Including constraints of the form: the same tag cannot appear in two disconnected segments n n n Eg: Title in a citation cannot appear twice Street name cannot appear twice Not relevant for named-entity tagging kinds of problems 25

Constrained Viterbi Original Viterbi Modified Viterbi …. 26

Comparative Evaluation n n Naïve model – One state per element in the HMM Independent HMM – One HMM per element; Rule Learning Method – Rapier Nested Model – Each state in the Naïve model replaced by a HMM 27

Results: Comparative Evaluation Dataset insta nces Elem ents IITB student 2388 Addresses 17 Company Addresses 769 6 US Addresses 740 6 The Nested model does best in all three cases (from Borkar 2001) 28

Results: Effect of Feature Hierarchy Feature Selection showed at least a 3% increase in accuracy 29

Results: Effect of training data size HMMs are fast Learners. We reach very close to the maximum accuracy with just 50 to 100 addresses 30

HMM approach: summary Inter-element sequencing ð Outer HMM transitions Intra-element sequencing ð Inner HMM Element length ð Multi-state Inner HMM Characteristic words ð Dictionary Non-overlapping tags ð Global optimization 31