Maximum Entropy Model Generalized Iterative Scaling Arindam Bose

Maximum Entropy Model & Generalized Iterative Scaling Arindam Bose CS 621 – Artificial Intelligence 27 th August, 2007

Statistical Classification o o o Statistical Classification problems Task is to find probability of class “a” occurring with context “b” or P(a, b) Context depends on nature of task n Eg. In NLP tasks, the context may consist of several words and associated syntactic labels

Training Data o o o Gather information from training data Large training data will contain some information about the co-occurrence of a’s and b’s Information never enough to completely specify P(a, b) for all possible (a, b) pairs

Task formulation o o o Find a method using the sparse evidence about a’s and b’s to reliably estimate a probability model, P(a, b) Principle of Maximum Entropy: the correct distribution P(a, b) is that which maximizes entropy, or “uncertainity” subject to constraints Constraints represent evidence

The Philosophy o o Making inferences on the basis of partial information without biasing the assignment that would amount to arbitrary assumptions of information that we do not have Maximize where remaining consistent with evidence

Representing evidence o o Encode useful facts as features and impose constraints on the expectations of these features A feature is a binary valued function, g(f, h)=1 if current_token_capitalized(h)=true and f=location_start =0 otherwise o Given k features, constraints have the form, i. e. the model’s expectation for each feature should match the observed expectation

Maximum Entropy Model o o Maximum entropy solution allows computation of P(f|h) for any f (a possible future/class) from every h (a possible history/context) “History” or the “context” is the conditioning data which enables the decision

The Model o o In the model produced by M. E. estimation, every feature has an associated parameter Conditional probability is calculated as : where o M. E estimation technique guarantees for every feature gi, the expected value of gi according to the M. E. model will equal the empirical expectation of gi in the training corpus

Generalized Iterative Scaling o o o Generalized Iterative Scaling, GIS, finds parameters of the distribution P GIS requires the constraint that If not, choose correctional feature, and add a such that

The GIS procedure It can be proven that a probability sequence whose parameters are defined by this procedure converges to a unique and positive solution

Computation o o o Each iteration requires computation of and Given training sample, , calculation of is straightforward The computation of the model’s feature expectation can be intractable

Computation of o We have, o Baye’s rule, o We use an approximation, summing over all the histories in the training sample, and not

Termination and Running Time o o Termination after a fixed number of iterations (e. g. 100) or negligible change in log likelihood Running time of each iteration dominated by computation of which is O(NPA)