Text Classification Elnaz Delpisheh Introduction to Computational Linguistics

Text Classification Elnaz Delpisheh Introduction to Computational Linguistics York University Department of Computer Science and Engineering 04 December 2020

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation Subjective text classification

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Text Classification-Definition Text classification is the assignment of text documents to one or more predefined categories based on their content. Text document Classifier Class A Text documen t Class B Class C Text documen t The classifier: Input: a set of m hand-labeled documents (x 1, y 1), . . , (xm, ym) Output: a learned classifier f: x y

Text Classification-Applications Classify news stories as World, US, Business, Sci. Tech, Sports, Entertainment, Health, Other. Classify business names by industry. Classify student essays as A, B, C, D, or F. Classify email as Spam, Other. Classify email to tech staff as Mac, Windows, . . . , Other. Classify pdf files as Research. Paper, Other Classify documents as Written. By. Reagan, Ghost. Written Classify movie reviews as Favorable, Unfavorable, Neutral. Classify technical papers as Interesting, Uninteresting. Classify jokes as Funny, Not. Funny. Classify web sites of companies by Standard Industrial Classification (SIC) code.

Text Classification-Example Best-studied benchmark: Reuters-21578 newswire stories ARGENTINE REGISTRATIONS 96031986/87 train, GRAIN/OILSEED 3299 test documents, 80 -100 words each, 93 BUENOS AIRES, Feb 26 classes Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: • Bread wheat prev 1, 655. 8, Feb 872. 0, March 164. 6, total 2, 692. 4 (4, 161. 0). • Maize Mar 48. 0, total 48. 0 (nil). • Sorghum nil (nil) • Oilseed export registrations were: • Sunflowerseed total 15. 0 (7. 9) • Soybean May 20. 0, total 20. 0 (nil) The board also detailed export registrations for subproducts, as follows. . Categories: grain, wheat (of 93 binary choices)

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Text Classification-Representing Texts f( ARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONS BUENOS AIRES, Feb 26 Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: • Bread wheat prev 1, 655. 8, Feb 872. 0, March 164. 6, total 2, 692. 4 (4, 161. 0). • Maize Mar 48. 0, total 48. 0 (nil). • Sorghum nil (nil) • Oilseed export registrations were: • Sunflowerseed total 15. 0 (7. 9) • Soybean May 20. 0, total 20. 0 (nil) The board also detailed export registrations for subproducts, as follows. . )=y simplest useful ? What is the best representation for the document x being classified?

Pre-processing the Text Removing stop words Punctuations, Prepositions, Pronouns, etc. Stemming Walk, walker, walked, walking. Indexing Dimensionality reduction

Representing text: a list of words f( ARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONS BUENOS AIRES, Feb 26 Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: • Bread wheat prev 1, 655. 8, Feb 872. 0, March 164. 6, total 2, 692. 4 (4, 161. 0). • Maize Mar 48. 0, total 48. 0 (nil). • Sorghum nil (nil) • Oilseed export registrations were: • Sunflowerseed total 15. 0 (7. 9) • Soybean May 20. 0, total 20. 0 (nil) The board also detailed export registrations for subproducts, as follows. . )=y f( (argentine, 1986, 1987, grain, oilseed, registrations, buenos, aires, feb, 26, argentine, grain, board, figures, show, crop, registrations, of, grains, oilseeds, and, their, products, to, february, 11, in, … )=y

Pre-processing the Text-Indexing Using vector space model

Indexing (Cont. )

Indexing (Cont. ) tfc-weighting It considers the normalized length of documents (M). ltc-weighting It considers the logarithm of the word frequency to reduce the effect of large differences in frequencies. Entropy weighting

Indexing-Word Frequency Weighting word ARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONS BUENOS AIRES, Feb 26 Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: Bread wheat prev 1, 655. 8, Feb 872. 0, March 164. 6, total 2, 692. 4 (4, 161. 0). • Maize Mar 48. 0, total 48. 0 (nil). • Sorghum nil (nil) • Oilseed export registrations were: • Sunflowerseed total 15. 0 (7. 9) • Soybean May 20. 0, total 20. 0 (nil) The board also detailed export registrations for subproducts, as follows. . • If the order of words doesn’t matter, x can be a vector of word frequencies. Categories: grain, wheat freq grain(s) 3 oilseed(s) 2 total 3 wheat 1 maize 1 soybean 1 tonnes 1 . . . “Bag of words”: a long sparse vector x=(, …, fi, …. ) where fi is the frequency of the i-th word in the vocabulary

Pre-processing the Text. Dimensionality Reduction Feature selection: It attempts to remove non- informative words. Document frequency thresholding Information gain Latent Semantic Indexing Singular Value Decomposition

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Text Classification-Methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Text Classification-Naïve Bayes Represent document x as list of words w 1, … For each class y, build a probabilistic model Pr(X|Y=y) of “documents” Pr(X={argentine, grain. . . }|Y=wheat) =. . Pr(X={stocks, rose, in, heavy, . . . }|Y=non. Wheat) =. . To classify, find the y which was most likely to generate x—i. e. , which gives x the best score according to Pr(x|y) f(x) = argmaxy. Pr(x|y)*Pr(y)

Text Classification-Naïve Bayes How to estimate Pr(X|Y) ? Simplest useful process to generate a bag of words: pick word 1 according to Pr(W|Y) repeat for word 2, 3, . . each word is generated independently of the others (which is clearly not true) but means How to estimate Pr(W|Y)?

Text Classification-Naïve Bayes How to estimate Pr(X|Y) ? Estimate Pr(w|y) by looking at the data. . . Terms: • This Pr(W|Y) is a multinomial distribution • This use of m and p is a Dirichlet prior for the multinomial

Text Classification-Naïve Bayes How to estimate Pr(X|Y) ? for instance: m=1, p=0. 5 This Pr(W|Y) is a multinomial distribution This use of m and p is a Dirichlet prior for the multinomial

Text Classification-Naïve Bayes Putting this together: for each document xi with label yi for each word wij in xi count[wij][yi]++ count++ to classify a new x=w 1. . . wn, pick y with top score: key point: we only need counts for words that actually appear in document x

Naïve Bayes for SPAM filtering

Naïve Bayes-Summary Pros: Very fast and easy-to-implement Well-understood formally & experimentally Cons: Seldom gives the very best performance “Probabilities” Pr(y|x) are not accurate e. g. , Pr(y|x) decreases with length of x Probabilities tend to be close to zero or one

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

The Curse of Dimensionality How can you learn with so many features? For efficiency (time & memory), use sparse vectors. Use simple classifiers (linear or loglinear) Rely on wide margins.

Margin-based Learning + + ++ + + + -- - - The number of features matters: but- not if the margin is sufficiently wide and examples are sufficiently close to the origin (!!)

Text Classification-Voted Perceptron Text documents: X=< x 1, x 2, … , xk > K= number of features Two classes={yes, no} W= < w 1, w 2, … , wk > Objective: Learn a weight vector W and a threshold θ If Yes Otherwise No If the answer is Correct: w 1 ++ Incorrect: There is a mistake. Correction is made: xk+1 = xk + xiwk k = k+1 wk+1 = 1

Voted Perceptron-Error Correction

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Lessons of the Voted Perceptron shows that you can make few mistakes in incrementally learning as you pass over the data, if the examples x are small (bounded by R), some u exists that has large margin. Why not look for this line directly? Support vector machines: • find u to maximize the margin.

Text Classification-Support vector Machines Facts about support vector machines: the “support vectors” are the xi’s that touch the margin. the classifier can be written as where the xi’s are the support vectors. support vector machines often + + ++ + + + - + give very good results on topical text classification. + - -- - -

Support Vector Machine Results

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Text Classification-Decision Trees Objective of decision tree learning Learn a decision tree from a set of training data The decision tree can be used to classify new examples Decision tree learning algorithms ID 3 (Quinlan, 1986) C 4. 5 (Quinlan, 1993) CART (Breiman, Friedman, et. al. 1983) etc.

Decision Trees-CART Creating the tree Splitting the set of training vectors The best splitter has the purest children Diversity measure is entropy Where S: set of example; k: number of classes; P(Ci): the probability of examples in S that belong to Ci. To find the best splitter each component of the document vector is considered in turn. This process is repeated until no sets can be partitioned any further. Each leaf is now assigned a class.

Decision Trees-Example

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

TF-IDF Representation The results above use a particular way to represent documents: bag of words with TFIDF weighting “Bag of words”: a long sparse vector x=(, …, fi, …. ) where fi is the “weight” of the i-th word in the vocabulary for word w that appears in DF(w) documents out of N in a collection, and appears TF(w) times in the documents being represented use weight: also normalize all vector lengths (||x||) to 1

TF-IDF Representation TF-IDF representation is an old trick from the information retrieval community, and often improves performance of other algorithms: K- nearest neighbor algorithm Rocchio’s algorithm

Text Classification-K-nearest Neighbor The nearest neighbor algorithm Objective To classify a new object, find the object in the training set that is most similar. Then assign the category of this nearest neighbor. Determine the largest similarity with any element in the training set: Collect the subset of X that has highest similarity with :

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Text Classification-Rocchio’s Algorithm classify using distance to centroid of documents from each class Assign test document to the class with maximum similarity

Support Vector Machine Results

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Text Classification-Neural Networks A neural network text classifier is a collection of interconnected neurons representing documents that incrementally learn from its environment to categorize the documents. A neuron training set X 1 W 1 X 2 W 2. . . Xn Wn y 1 U =∑Xi. Wi F(u) y 2 y 3

Text Classification-Neural Networks Forward and Backward propagation Forward propagation u 11 IM 1 H 1 OS 1(r) H 2 IM 2 vj. . . IM n uji . . . Hm Backward propagation OS 2(r) OS 3(r)

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation subjective text classification

Performance Evaluation Performance of a classification algorithm can be evaluated in the following aspects Predictive performance How accurate is the learned model in prediction? Complexity of the learned model Run time Time to build the model Time to classify examples using the model Here we focus on the predictive performance

Predictive Performance Evaluation How predictive is the model? Most common performance measures Classification accuracy on a dataset Recall Precision F-measure Fallout Accuracy Error … Classification error rate (1 -accuracy) Error on the training data is not a good indicator of performance on future data.

Performance Measures Correct Incorrect Assigned a b Rejected c d

Performance Measures(Cont. ) macro-averaging Gives equal weight to each category Computes the performance measures per category Computes the global means by averaging the above result micro-averaging Gives equal weight to each document Constructs a global contingency table Computes performance measure

Macro/Micro-averaging Example Class 1 Class 2 Yes No Call: yes 10 10 Call: no 10 970 Pooled table Yes No Call: yes 90 10 Call: no 10 890 Yes No Call: yes 100 20 Call: no 20 1860 Macro-averaged precision: [10/(10+10)+90/(10+90)]/2=(0. 5+0. 7)/2=0. 7 Micro-averaged precision: 100/(100+20)=0. 83

Estimating Error Rate Hold-out estimation Randomly split the data set into a training set and a test set, e. g. , training set (2/3), test set (1/3) Build model from the training set and estimate the error on the test set Repeat hold-out estimation Hold-out estimate can be made more reliable by repeating the process with different random splits. For each split, an error rate is collected. An overall error rate is obtained by averaging the error rates on the different splits.

Estimating Error Rate(cont. ) Cross-validation Randomly divide the data set into k subsets of equal size Use k-1 subsets as training data and one subset as test data- do this k times using each subset in turn for testing The error rates are averaged to yield an overall error estimate This is called k-fold cross-validation Standard method for evaluation: 10 -fold cross-validation Why 10? Extensive experiments have shown that this is best choice to get an accurate estimate.

Overfitting occurs when a model begins to memorize the training data rather than learning to generalize from the model. It can learn to perfectly predict the training data but it fails to predict the unseen data. Why? Insufficient examples Complex model Too many examples Noisy data

Underfitting occurs when a model is too simple to find the relationship in the input data. It fails to work well both on the training data and the unseen data.

Outline Definition and applications Representing texts Pre-processing the text Text classification methods Naïve Bayes Voted Perceptron Support Vector Machines Decision Trees K-nearest neighbor Rocchio’s algorithm Neural Networks Performance evaluation Subjective text classification

Text Classification: Examples Classify news stories as World, US, Business, Sci. Tech, Sports, Entertainment, Health, Other: topical classification, few classes Classify email to tech staff as Mac, Windows, . . . , Other: topical classification, few classes Classify email as Spam, Other: topical classification, few classes Adversary may try to defeat your categorization scheme Add Me. SH terms to Medline abstracts e. g. “Conscious Sedation” [E 03. 250] topical classification, many classes Classify web sites of companies by Standard Industrial Classification (SIC) code. topical classification, many classes Classify business names by industry. Classify student essays as A, B, C, D, or F. Classify pdf files as Research. Paper, Other Classify documents as Written. By. Reagan, Ghost. Written Classify movie reviews as Favorable, Unfavorable, Neutral. Classify technical papers as Interesting, Uninteresting. Classify jokes as Funny, Not. Funny.

Classifying Reviews as Favorable or Not Dataset: 410 reviews from Epinions Autos, Banks, Movies, Travel Destinations Learning method: Extract 2 -word phrases containing an adverb or adjective (eg “unpredictable plot”) Classify reviews based on average Semantic Orientation (SO) of phrases found: Computed using queries to web search engine

Classifying Reviews as Favorable or Not

Classifying Movie Reviews Idea: like Turney, focus on “polar” sections: subjective sentences

"Fearless" allegedly marks Li's last turn as a martial arts movie star--at 42, the ex-wushu champion-turned-actor is seeking a less strenuous on-camera life--and it's based on the life story of one of China's historical sports heroes, Huo Yuanjia. Huo, a genuine legend, lived from 1868 -1910, and his exploits as a master of wushu (the general Chinese term for martial arts) raised national morale during the period when beleaguered China was derided as "The Sick Man of the East. " "Fearless" shows Huo's life story in highly fictionalized terms, though the movie's most dramatic sequence--at the final Shanghai tournament, where Huo takes on four international champs, one by one--is based on fact. It's a real old-fashioned movie epic, done in director Ronny Yu's ("The Bride with White Hair") usual flashy, Hong Kong-and-Hollywood style, laced with spectacular no-wires fights choreographed by that Bob Fosse of kung fu moves, Yuen Wo Ping ("Crouching Tiger" and "The Matrix"). Dramatically, it's on a simplistic level. But you can forgive any historical transgressions as long as the movie keeps roaring right along.

"Fearless" allegedly marks Li's last turn as a martial arts movie star--at 42, the ex-wushu champion-turned-actor is seeking a less strenuous on-camera life--and it's based on the life story of one of China's historical sports heroes, Huo Yuanjia. Huo, a genuine legend, lived from 1868 -1910, and his exploits as a master of wushu (the general Chinese term for martial arts) raised national morale during the period when beleaguered China was derided as "The Sick Man of the East. " "Fearless" shows Huo's life story in highly fictionalized terms, though the movie's most dramatic sequence--at the final Shanghai tournament, where Huo takes on four international champs, one by one--is based on fact. It's a real old-fashioned movie epic, done in director Ronny Yu's ("The Bride with White Hair") usual flashy, Hong Kong-and-Hollywood style, laced with spectacular no-wires fights choreographed by that Bob Fosse of kung fu moves, Yuen Wo Ping ("Crouching Tiger" and "The Matrix"). Dramatically, it's on a simplistic level. But you can forgive any historical transgressions as long as the movie keeps roaring right along.

Classifying Movie Reviews [Pang et al, ACL 2004] Dataset: Rotten Tomatoes (+), IMDB plot reviews (-) Apply ML to build a sentence classifier Try and force nearby sentences to have similar subjectivity: use methods to find minimum cut on a constructed graph

Classifying Movie Reviews “subjective” One vertex for each sentence Confidence in classifications “non subjective” Edges indicate proximity

Classifying Movie Reviews [Pang et al, ACL 2004] Pick class + vs – for v 1 Pick class - vs + for v 2, v 3 Retained f(v 2)=f(v 3), but not f(v 2)=f(v 1)

Summary & Conclusions There are many, many applications of text classification In many applications, distinct classification decisions are interdependent Reviews: Subjectivity of nearby Topical classification is fairly well understood Most of the information is in individual words Very fast and simple methods work well In many applications, classes are not topics Sentiment detection Subjectivity/opinion detection sentences Lots of prior work to build on, lots of prior experimentation to consider

References K. Aas, L. Eikvil, Text Categorization: A Survey, June 1999. W. W. Cohen, Text Classification: An Advanced Tutorial. Machinelearning Department, Carnegie Mellon, 2010. Manning, C. D. , and Schutze, H. (1999). Foundations of Statistical Natural Language Processing, MIT Press, Cambridge, MA. A. C. Cachopo, A. L. Oliveira, An Emprical Comparison of Text Categorization Methods, String Processing and Information Retrieval, 10 th International Symposium, Springer Verlag, 2003. C. Donghui, L. Zhijing, A New Text Categorization Method based on HMM and SVM, International Symposium on Communications and Information Technologies, 2007. W. Xue, X. Xu, Three New Feature Weighting Methods for Text categorization, Springer-Verlag Berlin Heidelberg

References(Cont. ) F. Xia, T. Jicun, L. Zhihui, A Text Categorization Method Based on Local Document Frequency, IEEE Computer Society, 2009. F. Sebastiani, Machine Learning in Automated text Categorization, ACM Computing Surveys, 2002. K. Nigam, A. K. Mccallum, S. Thrun, T. Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Special issue on information retrieval ACM, 2000. M. E. Ruiz, P. Srinivasan, Hierarchical Text Categorization using Neural Networks, Kluwer Academic Publications, Springer, 2002.

References(Cont. ) J. Meng, H. Lin, A Two-stage Feature Selection Method for Text Categorization, Seventh International Confeence on Fuzzy Systems and Knowledge Discovery, IEEE, 2010.

Thanks!