Loglinear Models Discriminative Model and CRF Conditional Random

Log-linear Models, Discriminative Model and CRF (Conditional Random Field) SUBJECT: NATURAL LANGUAGE PROCESSING NAME: MAHLAKA ABBAS ROLL NO: 19 MESE 07

Log Linear Models

The General Problem

Language Modeling Consider again the language modeling problem, where § The input x is a sequence of words w 1, w 2, ………wi-1, and § The label y is a word wi so P(y|x) = (wi | w 1, w 2 ……. wi-1)

Feature Vector

Log-Linear Models

Features for the Language Modeling Example The first three features, f 1; f 2, and f 3, are analogous to unigram, bigram, and trigram features in a regular trigram language model. Ø The first feature returns 1 if the label y is equal to the word model, and 0 otherwise. Ø The second feature returns 1 if the bigram model, and 0 otherwise. ØThe third feature returns 1 if the trigram model, is equal to statistical is equal to any statistical

Example (Conti. . )

Generative Vs Discriminative Models

Generative Vs Discriminative Models A Generative Model learns the joint probability distribution p(x, y). It predicts the conditional probability with the help of Bayes Theorem. A Discriminative model learns the conditional probability distribution p(y|x). Both of these models were generally used in supervised learning problems.

Generative Vs Discriminative Models (Conti. . )

Generative Vs Discriminative Models (Conti. . ) Training classifiers involve estimating f: X Y, or P(Y|X) Generative classifiers - Assume some functional form for P(Y), P(X|Y) - Estimate parameters of P(X|Y), P(Y) directly from training data - Use Bayes rule to calculate P(Y |X) Discriminative Classifiers - Assume some functional form for P(Y|X) - Estimate parameters of P(Y|X) directly from training data

Generative Vs Discriminative Models (Conti. . ) Generative classifiers q Naive Bayes q Hidden Markov Models (HMM) Discriminative Classifiers q Log Linear Model q Maximum Entropy Markov Model (MEMM) q Conditional Random Fields (CRF)s

Conditional Random Field

Conditional Random Field Ø Conditional Random Field (CRF) is intended to do the task-specific predictions i. e. we have the input X (vector) and predict the label y which are predefined. Ø CRF is a probabilistic discriminative model that has a wide range of applications in Natural Language Processing and Computer Vision ØThe conditional random field is used for predicting the sequences that use the contextual information to add information which will be used by the model to make correct predictions.

Conditional Random Field (Conti. . ) One of the famous application of CRFs in NLP is Named Entity Recognition (NER) where we predict the sequence in which they are dependent on each other. There are other various types of CRFs such as Ø Hidden CRF used for gesture recognition, Ø Dynamic CRF for labeling sequence data, Ø Skip Gram CRF for activity recognition and so on.