Tackling the Poor Assumptions of Naive Bayes Text

Tackling the Poor Assumptions of Naive Bayes Text Classifiers Pubished by: Jason D. M. Rennie, Lawrence Shih, Jamime Teevan, David R. Karger Liang Lan 11/19/2007

Outline Introduce the Multinomial Naive Bayes Model for Text Classification. n The Poor Assumption of Multinomial Naive Bayes Model. n Solutions to some problem of the Naive Bayes Classifier. n

Multinomial Naive Bayes Model for Text Classification n Given: ¨A description of the document d: f = (f 1, …. . , fn) fi is the frequency count of word i occurring in document d ¨ A fixed number of classes: C = {1, 2, …, m}, Parameter Vector for each class The parameter vector for a class c is θ n ci is the probability of word i occurs in class c Determine: ¨ The class label of d.

Introduce the Multinomial Naive Bayes Model for Text Classification n The likelihood of a document is a product of the parameters of the words that appear in the document. n Selecting the class with the largest posterior probability

Parameter Estimation for Naive Bayes Model n The parameters θci must be estimated from the training data. n Then, we get the MNB classifier. n For simplicity, we use uniform prior estimate. l. MNB(d) = argmaxc(fiwci) n

The Poor Assumption of Multinomial Naive Bayes Model Two systemic errors (Occurring in any naive bayes classifier ) 1. Skewed Data Bias ( uneven training size) 2. Weight Magnitude Errors (Caused by the independence assumption) n The Multinomial does not model the text well n

Correcting the skewed data bias More training examples for one class than another --- can cause the classifier to prefer one class over the other. n Using Complement Naive Bayes n n N~ci is the number of times word i occurred in documents in classes other than c.

Correcting the Weight Magnitude Errors n n Caused by the independence assumption Ex. “San Francisco” , “Boston” Normalizing the Weight Vectors We call this Weight-normalized Complement Naive Bayes(WCNB).

Modeling Text Better Transforming Term Frequency n Transforming by Document Frequency n Transforming Based on Length n

Transforming Term Frequency n The term distribution had heavier tails than predicted by the multinomial model, instead appearing like a power-law distribution. The probability is also proportional to So we can use the multinomial model to generate probabilities proportional to a class of power law distribution via a simple transform,

Transforming by Document Frequency Common words are unlikely to be related to the class of a document, but random variations can create apparent fictitious correlation. n Discount the weight of the common words. Inverse document frequency (a common IR transform)– to discount terms by their document frequency. n

Transforming Based on Length The jump for larger term frequency is disproportionally large with the length of the document. n Discount the influence of long documents by transforming the term frequency: n

The New Naive Bayes procedure

The result of experiment comparing MNB to TWCNB and the SVM shows that the TWCNB’s performance is substantially better than MNB, and approach the SVM’s performance.