Special topics on text mining Part I text

  • Slides: 30
Download presentation
Special topics on text mining [Part I: text classification] Hugo Jair Escalante, Aurelio Lopez,

Special topics on text mining [Part I: text classification] Hugo Jair Escalante, Aurelio Lopez, Manuel Montes and Luis Villaseñor

Multi label text classification Hugo Jair Escalante, Aurelio Lopez, Manuel Montes and Luis Villaseñor

Multi label text classification Hugo Jair Escalante, Aurelio Lopez, Manuel Montes and Luis Villaseñor Most of this material was taken from: G. Tsoumakas, I. Katakis and I. Vlahavas. Mining multi-label data. Data Mining and Knowledge Discovery Handbook, Part 6, O. Maimon, L. Rokach (Ed. ), Springer, 2 nd edition, pp. 667685, 2010.

Machine learning approach to TC • Develop automated methods able to classify documents with

Machine learning approach to TC • Develop automated methods able to classify documents with a certain degree of success Trained machine Training documents (Labeled) Learning machine (an algorithm) ? Labeled document Unseen (test, query) document

What is a learning algorithm? • A function: • Given:

What is a learning algorithm? • A function: • Given:

Binary vs multiclassification • Binary classification: each document can belong to one of two

Binary vs multiclassification • Binary classification: each document can belong to one of two classes. • Multiclassification: each document can belong to one of K classes.

Classification algorithms • (Some) classification algorithms for TC : – Naïve Bayes Some of

Classification algorithms • (Some) classification algorithms for TC : – Naïve Bayes Some of this – K-Nearest Neighbors methods were – Centroid-based classification designed for binary classification – Decision trees problems – Support Vector Machines – Linear classifiers (including SVMs) – Boosting, bagging and ensembles in general – Random forest – Neural networks

Linear models • Classification of DNA micro-arrays x 2? Cancer ? No Cancer x

Linear models • Classification of DNA micro-arrays x 2? Cancer ? No Cancer x 1

Main approaches to multiclassification • Single machine: Learning algorithms able to deal with multiple

Main approaches to multiclassification • Single machine: Learning algorithms able to deal with multiple classes (e. g. , KNN, Naïve Bayes) • Combining the outputs of several binary classifiers: – One-vs-all: one classifier per-class – All-vs-all: one classifier pair of classes

Multilabel classification • To what category belong these documents:

Multilabel classification • To what category belong these documents:

Multilabel classification • A function: • Given:

Multilabel classification • A function: • Given:

Conventions n xi a X={xij} m y ={yj} w Slide taken from I. Guyon.

Conventions n xi a X={xij} m y ={yj} w Slide taken from I. Guyon. Feature and Model Selection. Machine Learning Summer School, Ile de Re, France, 2008.

Conventions |L| n xi a X={xij} m Z ={Zj} w Slide taken from I.

Conventions |L| n xi a X={xij} m Z ={Zj} w Slide taken from I. Guyon. Feature and Model Selection. Machine Learning Summer School, Ile de Re, France, 2008.

Multi-label classification • Each instance can be associated to a set of labels instead

Multi-label classification • Each instance can be associated to a set of labels instead of a single one • Specialized multilabel classification algorithms must be developed • How to deal with the multilabel classification problem?

(Text categorization is perhaps the dominant multilabel application)

(Text categorization is perhaps the dominant multilabel application)

Multilabel classifiers • Transformation methods: Transform the multilabel classification task into several single -label

Multilabel classifiers • Transformation methods: Transform the multilabel classification task into several single -label problems • Adaptation approaches: Modify learning algorithms to support multilabel classification problems

Transformation methods • Copy transformation. Transforms the multilabel instances into several single-label ones Original

Transformation methods • Copy transformation. Transforms the multilabel instances into several single-label ones Original ML problem Transformed ML problem (unweighted) Transformed ML problem (weighted)

Transformation methods • Select transformation. Replaces the multilabel of each instance by a single

Transformation methods • Select transformation. Replaces the multilabel of each instance by a single one Max Original ML problem Min Rand Transformed ML problem Ignore approach

Transformation methods • Label power set. Considers each unique set of labels in the

Transformation methods • Label power set. Considers each unique set of labels in the ML problem as a single class Original ML problem Transformed ML problem Pruning can be applied

Transformation methods • Binary relevance. Learns a different classifier per each different label. Each

Transformation methods • Binary relevance. Learns a different classifier per each different label. Each classifier i is trained using the whole data set by considering examples of class i as positive and examples of other classes (j≠i) as negative Original ML problem Data sets generated by BR • How labels are assigned to new instances?

Transformation methods • Ranking by pairwise comparison. Learns a different classifier per each pair

Transformation methods • Ranking by pairwise comparison. Learns a different classifier per each pair of different labels. Original ML problem Data sets generated by BR

Algorithm adaptation techniques • Many variants, including – Decision trees – Boosting ensembles –

Algorithm adaptation techniques • Many variants, including – Decision trees – Boosting ensembles – Probabilistic generative models – KNN – Support vector machines

Algorithm adaptation techniques • MLk. NN. For each test instance: – Retrieve the top-k

Algorithm adaptation techniques • MLk. NN. For each test instance: – Retrieve the top-k nearest neighbors to each instance – Compute the frequency of occurrence of each label – Assign a probability to each label and select the labels for the test instance

Feature selection in multilabel classification • An (almost) unstudied topic = opportunities • Wrappers

Feature selection in multilabel classification • An (almost) unstudied topic = opportunities • Wrappers can be applied directly (define an objective function to optimize based on a multilabel classifier) Original feature set Generation Subset of feature no Evaluation Stopping criterion Validation yes Selected subset of feature process From M. Dash and H. Liu. http: //www. comp. nus. edu. sg/~wongszec/group 10. ppt

Feature selection in multilabel classification • An almost un-studied topic = opportunities • Existing

Feature selection in multilabel classification • An almost un-studied topic = opportunities • Existing filter methods transform the multilabel problem and apply standard filters for feature selection

Statistics • Label cardinality • Label density

Statistics • Label cardinality • Label density

Evaluation of multilabel learning • (New) conventions: Data set Labels Predictions of a ML

Evaluation of multilabel learning • (New) conventions: Data set Labels Predictions of a ML classifier for instances in D

Evaluation of multilabel learning • Hamming loss: • Classification accuracy:

Evaluation of multilabel learning • Hamming loss: • Classification accuracy:

Evaluation of multilabel learning • Precision: • Recall:

Evaluation of multilabel learning • Precision: • Recall:

Evaluation of multilabel learning • F 1 -measure

Evaluation of multilabel learning • F 1 -measure

Suggested readings • G. Tsoumakas, I. Katakis, I. Vlahavas. Mining multi-label data. Data Mining

Suggested readings • G. Tsoumakas, I. Katakis, I. Vlahavas. Mining multi-label data. Data Mining and Knowledge Discovery Handbook, Part 6, O. Maimon, L. Rokach (Ed. ), Springer, 2 nd edition, pp. 667 -685, 2010. • G. Tsoumakas, I. Katakis. Multi-label classification: an overview. International Journal of Data Warehousing, 3(3), 1— 13, 2007. • M. Zhang, Z. Zhou. ML-k. NN, A lazy learning approach to multi-label learning. Pattern recognition 40: 2038 — 2048, 2007. • M. Boutell, J. Luo, X. Shen. C. Brown. Learning multi-label scene classification. Pattern recognition 37: 1757— 1771, 2004.