Text Classification from Labeled and Unlabeled Documents using

Text Classification from Labeled and Unlabeled Documents using EM - Kamal Nigam Andrew Kachites Mccallum Sebastian Thrun Tom Mitchell Presented by Yuan Fang, Fengyuan Hu and Sandhya Prabhakaran

Job Hunting?

Roadmap ¡ Part 1 – Text Classification ¡ Part 2 – Incorporating Unlabeled data with EM ¡ Part 3 – Results and Recap

Part I Text Classification

Text Classification – the Definition ¡ “Text classification systems categorize documents into one (or several) of a set of pre-defined topics of interest”

How Are Automatic Text Classifiers Created ¡ Before: Manual construction of rule sets (Painful and time-consuming ) ¡ Present : Supervised learning to construct a classifier (efficient and successful)

What To Provide ¡ An algorithm with an example set of documents for each class and allow it to find a representation or decision rule for classifying future documents automatically ¡ This approach will : - give high-accuracy classifiers - be significantly less expensive

What Data is Available ¡ Key difficulty : A large number of labeled training examples are required to learn accurately - What we need but don't have ¡ One would obviously prefer algorithms that can provide accurate classifications after hand labeling only a dozen articles, rather than thousands ¡ What other sources of information can reduce the need for labeled data?

Unlabeled data ¡ How unlabeled data can be used to increase classification accuracy, especially when labeled data are scarce ¡ An intuitive example

Goal And Merit ¡ The goal – To demonstrate that supervised learning algorithms can use a small number of labeled examples with a large number of unlabeled examples to create highaccuracy text classifiers ¡ The merit – Unlabeled examples are much less expensive and easily available

Parametric Generative Model Overview ¡ Assumption : a statistical process generates the documents (words and class labels) ¡ statistical process - parametric generative model

Incorporating Unlabeled Data with Generative Models ¡ Using EM to find high-probability parameters of the model given a combination of labeled and unlabeled data ¡ Experimental evidence shows that using unlabeled data with EM can increase classification accuracy

Assumptions In the Model (1) Documents are generated by a mixture of multinomials model, where each mixture component corresponds to a class (1 class to 1 component) (2) The mixture components are multinomial distributions of individual words - the words are produced independently of each other given the class

Two Multisided Dies ¡ Let there be |C| classes and a vocabulary of size |V|; each document d has |d| words in it. ¡ First, we roll a biased |C|-sided die to determine the class of our document. ¡ We roll the biased |V|-sided die that corresponds to the chosen class |d| times and write down the indicated words. These words form the generated document.

Parametric Generative Model ¡ ¡ ¡ - parameters for the mixture model - mixture of components - mixture weights or class probabilities - document distribution of selected class Equation (1)

Denotation ¡ ¡ - the jth mixture component, as well as the jth class. - the class label for a particular document ( ¡ A document ¡ We write for the word - a word in the vocabulary ¡ ¡ ) is considered to be an ordered list of word events, in position k of - document length, chosen independently of the component, its own probability

Parametric Generative Model ¡ Expanding the Equation (1) with document length and the words in the document. Equation (2) ¡ The words of a document are generated independently of context Equation (3) ¡ Combining these last two equations gives the naive Bayes expression for the probability of a document given its class Equation (4)

Model Parameters ¡ Collection of word probabilities, each written ¡ ¡ Document length is identically distributed, no need to be parameterized for classification denoted as the mixture weights (class probabilities) ¡ The complete collection of model parameters

Naive Bayes Text Classification ¡ Using a collection of labeled documents for training ¡ Finding the most probable parameters for the statistical model introduced

Training A Naive Bayes Classifier With Labeled Data ¡ Estimating the parameters of the generative model by using a set of labeled training data (the estimate of the parameters is written ) ¡ Finding (MAP), the value of that is most probable given the evidence of the training data and a prior.

Training A Naive Bayes Classifier With Labeled Data ¡ The word probability estimates Equation (6) ¡ Class probabilities Equation (7) are given by

Classifying New Documents with Naive Bayes Equation (8) ¡ If the task is to classify a test document into a single class, then the class with the highest posterior probability is selected.

Part Ⅱ Incorporating Unlabeled Data with EM

The Problem ¡ ¡ ¡ The case that given only labeled data is explained already. MAP– to maximize the posterior probability. Naïve Bayes– do classification of labeled data. Now the case is given both labeled and unlabeled data. Searching for a solution? – Here it is!

Revision of EM ¡ ¡ ¡ Recall the EM knowledge in PMR – Might be painful, but helpful Mixture Model Hidden variable – z to active the components

Revision of EM ¡ ¡ ¡ EM applied to Gaussian Mixture Model Maximum Likelihood Estimation Parameters: µ andΣ E step: evaluate the responsibilities using current estimators/parameters M step: re-estimate by using the maximum a posteriori parameter Run the demo

Back to the paper

Back to the paper ¡ ¡ Collection of labeled and unlabeled documents. MAP Try to maximize P(θ|D) Bayesian method -- P(θ|D) → P(θ) P(D| θ)

Back to the paper ¡ ¡ Log likelihood Incomplete equation

Back to the paper ¡ ¡ z – binary indicator variables which is set to be 1 if y = c, else zero. Then problem of the incomplete log probability can be transferred to complete log probability of parameters.

Back to the paper ¡ ¡ ¡ Methods used in the paper Basic EM Augmented EM (1) Weighting the unlabeled data (2) Multiple mixture components per class

Basic EM ¡ ¡ Initialize the NB classifier using MAP parameter estimation, from only labeled dataset. E step: estimate the component membership by calculating its expected value generated by from only unlabeled data. M step: re-estimate the classifier for the whole data set, using MAP, loop from E step: Look at to measure the improvement of the parameters, decide when to stop the loop

Restrictions of Basic EM ¡ ¡ ¡ Assumptions/Restrictions: Large unlabeled data set, small labeled data set → if not true, unlabeled data will hurt the accuracy. One-to-one correspondence of components and classes → not so accurate because subtopics exist.

Augmented EM – weighting unlabeled data ¡ ¡ Method: weakening the contribution of unlabeled data while the labeled set is already good enough for classification. Equation:

Augmented EM – weighting unlabeled data ¡ λis decided by leave-one-out cross validation. is defined to tell whether it is labeled or unlabeled. ¡ Modified MAP parameters: ¡

Augmented EM -- multiple mixture components per class ¡ ¡ Method: Relax the assumption that one-to-one correspondence of components and classes. Many-to-one relationship between components and classes.

Augmented EM – multiple mixture components per class ¡ ¡ ¡ How? Decide the number of components per class by again crossvalidation. Mapping from components to classes:

The complete algorithm ¡ ¡ ¡ ¡ Collections of labeled, unlabeled documents. Set λby cross-validation. Set the number of components per class. Randomly assign for mixture components. Initialize the parameters θ of NB classifier using MAP. Loop until complete log likelihood of labeled and unlabeled data is satisfying enough. E step: estimate the component membership of each doc using θ M step: re-estimate θgiven the membership, still MAP.

Comparison ¡ ¡ ¡ Basic EM: performs well comparing with naïve bayes classifier alone, with large unlabeled dataset and small set of labeled data EM-λ: can apparently improve the accuracy if the assumption above doesn’t fit. Multiple Components: dramatically outperforms than basic EM.

Part III Results and Recap

Experimental Results ¡ Empirical evidence that on combining labeled with unlabeled data using EM outperforms naive Bayes. ¡ 20 Newsgroups, Web. KB, Reuters ¡ Improvements in accuracy due to unlabeled data are dramatic, especially when the number of labeled data is low. ¡ Augmented EM can increase performance even when basic EM performs poor due to large number of unlabeled data.

Data sets and Protocols - 20 Newsgroup ¡ 20017 articles divided evenly among 20 different Use. Net discussion groups. Task - to classify an article into the one newsgroup to which it was posted. Many categories fall into confusable clusters. Stop words are removed – 62258 unique words Word counts are normalized and scaled – each document has constant length. ¡ ¡

Data sets and Protocols - Web. KB ¡ ¡ ¡ 8145 Web pages gathered from university computer science departments. Choosing 4199 pages covering categories: student, faculty, course and project. Task - to classify a web page into one of the four categories. Stemming and stoplist are not used. Vocabulary is limited to 300 most informative words using leave-one-out cross validation.

Data sets and Protocols - Reuters ¡ 12902 articles and 90 topic categories. Task - to build a binary classifier for each of the ten most populous classes to identify the news topic. Words inside <TEXT> tags are used – REUTERS and &# not used. Stoplist are used, but no stemming. Metrics are Recall and Precision instead of Accuracy. ¡ ¡

Precision-Recall breakeven point • Standard information retrieval measure • Recall – number of correct positive predictions number of positive examples • Precision - number of correct positive predictions number of positive predictions

Wall-clock timing ¡ EM usually converges after 10 iterations ¡ Less than 1 minute for the Web. KB ¡ Less than 15 minutes for 20 Newsgroups – huge vocabulary and more documents

EM with unlabeled data increases Accuracy Figure 1: - Accuracy versus # of Labeled Documents. (20 Newsgroups)

Effect of varying the # of unlabeled documents Figure 2: - Accuracy versus # of unlabeled documents. (20 Newsgroups)

EM algorithm in action Figure 3: - ‘Course’ class for Web. KB dataset

EM performance degradation Figure 4: - As # of Labeled data increases, accuracy of classifier falls with more # of unlabeled data. Importance of weighting factor λ. (Web. KB)

Effects of different EM Figure 5: - Comparison between EM, CV EM-λ and EM-λ (Web. KB)

Performance of EM on different # of mixture components Figure 6: - Too few or too many mixture components result in poor performance. Unlabeled data is used. (Reuters)

Precision-Recall breakeven points Figure 7: - Comparison between NB and EM on Reuters dataset

Related Work ¡ ¡ ¡ EM is a well-known family of algorithms that works by treating unclassified data as incomplete. According to Miller et al - EM on non-textual tasks using mixture of Gaussians – assumed unlabeled data to be sufficient to estimate parameter values. Castelli and Cover - unlabeled data does not improve the classification results in the absence of labeled data. EM can be combined with active-learning to improve performance – now only slightly more than half of labeled data was enough! EM can be applied with other machine learning algorithms like SVM, k. NN.

Punchwords • Text classification • Naive Bayes • Expectation Maximisation Algorithm • EM-λ • Multiple Mixture models for subclass • Leave-one-out cross validation • Stemming and stoplist words • Accuracy, Precision, Recall

Recap ¡ ¡ ¡ A family of algorithms have been presented to address text classification using voluminous unlabeled data and scarce labeled data. When data is consistent with the assumptions - Basic EM performs well. When data is not consistent - 2 extensions hold valid - EM-λ: controlling the contribution of unlabeled data. - Multiple Mixture Components per Class: “many-to-one” constraint.

References ¡ Using Unlabeled Data to Improve Text Classification – ¡ ¡ May 2001 at www. kamalnigam. com/papers/thesis-nigam. pdf Netlab toolkit - www. ncrg. aston. ac. uk/netlab/ Validation Lecture – Intelligent Sensor Systems, Ricardo. Gutierrez-Osuna, Wright State University

Question Time!! Route further questions to. . . Ryan - 0789317 Neo - 0785401 Sandhya - 0671562 ¡ Thank you !!