Unsupervised Models for Named Entity Classification Michael Collins

Unsupervised Models for Named Entity Classification Michael Collins and Yoram Singer Yimeng Zhang March 1 st, 2007

Overview • Unlabeled data can be used to reduce the need for supervision • Basic idea: making use of the redundancy in the unlabeled data – Co. Training [ Blum and Mitchell 98] • Two unsupervised models – DL-Co. Train, based on Decision List Learning – Co. Boost, based on Ada. Boost algorithm

Problem setting Given • iid labeled examples – 7 seed rules • iid unlabeled examples – 90, 000 unlabeled examples • xi is a feature vector drawn from a set of possible values X Wish to learn a classification function f : X ->Y {Location, Person, Organization}

Redundantly Sufficient Features person spelling feature context feature Redundantly Sufficient Features: • features can be separated into two types • either X 1 or X 2 is sufficient for classification there exists functions f 1 and f 2 for any example , • X 1 and X 2 are conditionally independent given Y

Co. Training spelling context Mr. Cooper … a president of Each unlabeled pair is represented as an edge An edge indicates that the two features must have the same label

Co. Training spelling context Mr. Cooper … president Each unlabeled pair is represented as an edge An edge indicates that the two features must have the same label

Co. Training Given • Labeled examples • Unlabeled examples • xi = Induce functions f 1 and f 2 such that Loose the constraint

Supervised Algorithm based on Decision Lists • Input – xi is a set of features Decision List h(x 1, 0) h(x 2, 1) x 1 false x 2 false x 3 true 0 1 true • Output 0 – A function – h(x, y) is an estimate of the probability p(y|x) of seeing label y given that feature x is present – h can be thought of as defining a decision list of rules x->y ranked by their strength h(x, y) 1

Supervised Algorithm based on Decision Lists (2) • The label for a test example x Decision List h(x 1, 0) x 1 true h(x 2, 1) • h(x, y) is defined as follows false x 2 false x 3 0 – Count(x, y) is the number of times feature x is seen with label y in training data true 0 1 1

DL-Co. Train (unsupervised decision list) Initialize spelling rules x 1, s->y x 2, s->y x 3, s->y x 4, s->y label data context rules x 1, c->y x 2, c->y x 3, c->y spelling rules x 1, s->y x 2, s->y x 3, s->y x 4, s->y x 5, s ->y x 4, s->y label data induce rules Spelling rules … context rules … … iteration labeled data Induce rules: choose the rules with the features that appeared more times with some known label

Boosting-based algorithm - Ada. Boost D is a distribution over instances, specifies the relative weight of each example weight for the learner The training error is bounded above by Choose ht and the weight to minimize Zt

Ada. Boost for named entity recognition Weak hypothesis choose ht , so that it minimize

Co. Boost (unsupervised Ada. Boost) Recall the criteria for Co. Training Given • Labeled examples • Unlabeled examples • xi = (spelling, context) = Induce functions f 1 and f 2 such that

Co. Boost (2) Optimization function : a extension of Zt , learn f 1 and f 2 choose ht and to minimize this function error for labeled data the number of disagreements on unlabeled data is the unthresholded hypothesis for fj

Co. Boost (3) At each iteration • step 1: fix the second one, choose • step 2: fix the first one, choose and to minimize the first one to minimize the second one

Co. Boost (4) t is iteration, j is the classifier take the current output of the other classifier for unlabeled data the instance weight is based on this classifier the same form as the function Zt in Ada. Boost use the same algorithm as Ada. Boost to choose and

Evaluation 88, 962 examples (spelling, context) pairs 7 seed rules are used 1000 examples are chosen as test data. (85 noise) We label the examples to ( location, person, organization, noise)