Introduction to data mining G Marcou Laboratoire dinfochimie

Introduction to data mining G. Marcou+ +Laboratoire d’infochimie, Université de Strasbourg, 4, rue Blaise Pascal, 67000 Strasbourg 1

Motivation of data mining § Discover automatically useful information in large data repository. § Extract patterns from experience. § Predict outcome of future observations. § Learning: Experience Set of Task Preformance Measure If Experience increase, performance measure on the set of tasks increases

Organisation of data § Datasets are organized as instances/attributes Instances Synonyms Data points Entries Sample … Attributes Synonyms Factors Variables Measures. . .

Nature of data § Attributes can be: Atom counts: O=1 Cl=4 N=6 S=3 Numeric Nominal Molecule name: (1 -methyl)(1, 1, 1 -tributyl)azanium, tetrahexylammonium Molecular surface: Continuous Phase state: solid, amorphous, liquid, gas, ionized Categorical Ordered Intestinal absorption: not absorbed, mildly absorbed, completely absorbed Spectral domains: Ranges UV visible IR EC 1. Oxidoreductases Hierarchical EC numbers: EC 2. Transferases EC 6. 1 Forming Carbon-Oxygen Bonds EC 3. Hydrolases EC 4. Lyases EC 5. Isomerases EC 6. Ligases EC 6. 2 Forming Carbon-Sulfur Bonds EC 6. 3 Forming Carbon-Nitrogen Bonds EC 6. 4 Forming Carbon-Carbon Bonds EC 6. 5 Forming Phosphoric Ester Bonds EC 6. 6 Forming Nitrogen—Metal Bonds

Nature of learning § Unsupervised learning Clustering Rules + § Supervised learning Classification Regression § Other Reinforcement First order logic

Concept in data mining § A Concept is the target function to be learned § Concept is learned from Instance 1 attributes-values Relations Sequences Spatial Instance 2 Instance 3 … DB 1 DB 2

Machine Learning and Statistics § Data miner point of view Any hypothesis compatible with the dataset is useful Search for all hypothesis compatible with the dataset Induction § Statistician point of view Datasets are the expression of underlying probability distributions Datasets validate or invalidate prior hypothesis Deduction

Validation in Data Mining § Validation means that a model is build on a training set of data then applied on a test set of data. § Success and failure on the test set must be estimated. § The estimate is supposed to be representative of any new situation. § Every model must be validated.

Training/Test § Split the dataset in two parts: One part is the training set The other is the test set

Bootstrapping § Draw N instances with replacement from the dataset § Create a training set with these instances § Use the dataset as the test set

Cross-Validation § Split the dataset in N subsets § Use each subset as a test set while all others form a training set

Scrambling § Reassign at random the classes to the instances. § Success and failure are estimated on the scrambled data. § The goal is to estimate good success measurement by pure chance.

Clustering § Search for an internal organization of the data § Optimizes relations between instances relative to an objective function § Typical objective functions: Separation Contiguity Coherence Concept Density

Cluster Evaluation § Essential because any dataset can be clustered by not any cluster is meaningful. § Evaluation can Unsupervised Supervised Relative

Unsupervised Cluster evaluation Cohesion Separation Silhouette Clustering Tendency Proximity matrix Co. Phenetic Correlation For p Nearest Neighbor Distances (NND) between instances (ωi) and NND between rand points (ui)

Supervised cluster evaluation Cluster 3 Class 1 Precision(3, 1) Recall(3, 1) Precision(i, j) pij Ni, number of members of cluster i Recall(i, j)

Relative analysis § Compare two clustering. § Supervised cluster analysis is a special case of relative analysis The reference clustering is the set of classes Rand statistics Jaquard statistics N 00: number of instances couple in different clusters for both clustering N 11: number of instances couple in same clusters for both clusters N 01: number of instances couple in different clusters for the first clustering and in the same clusters for the second N 10: number of instances couple in the same clusters for the first clustering and in different one for the second.

A simple clustering algorithm: k-mean 1. Select k points as centroids 2. Form k clusters: each point is assigned to its closest centroid 3. Reset each centroid to the (geometric) center of its cluster 4. Repeat from point 2 until no change is observed 5. Repeat from point 1 until stable average clusters are obtained. X X X

Classification § Definition Assign one or several objects to predefined categories The target function maps a set of attributes x to a set of classes y. § Learning scheme Supervised learning Attribute-value § Goal Predict the outcome of future observations

Probabilities basics § Conditional probabilities Independence of random events: Probability of realization of event A knowing that B has occurred The Bayes equation for independent events xi

Statistical approach to classification Class 2 Class 1 § Estimate the probability of an instance {x 1, x 2} being of Class 1 or Class 2.

The Naive Bayes assumption § The probability that an instance {x 1, x 2, …} belongs to class A is difficult to estimate. Poor statistics § Consider the Bayes Equation: With the naive assumption that {x 1, x 2, …} are independent Likelihood Posterior Probability Prior Probability Evidence § The prior probability, the evidence and the likelihood have better estimates Good statistics

The Naive Bayes Classifier 1. Estimate the prior probability, P(A), for each class. 2. Estimate the likelihood, P(x|A), of each attribute for each class. 3. For a new instance, estimate the Bayes Score for each class: 4. Assign the instance to the class which possesses the highest score • The value of C can be optimized

Success and failure § For N instance and a give classifier, for each class I § NTP(i): True Positives • Number of instances of class i correctly classified. § NFP(i): False Positives • Number of instances incorrectly assigned to class i. § NTN(i): True Negatives • Number of instances of other classes correctly classified. § NFN(i): False Negatives • Number of instances of class i incorrectly assigned to other classes.

Confusion Matrix § For N instances, K classes and a classifier § Nij, the number of instances of class i classified as j Class 1 Class 2 … Class. K Class 1 N 12 … N 1 K Class 2 N 21 N 22 … N 2 K … … … Class. K NK 1 NK 2 … NKK

Classification Evaluation Global measures of success Measures are estimated on all classes Local measures of success Measures are estimated for each class

Ranking success evaluation Recall § Receiver Operator Curve (ROC) § Receiver Operator Curve Area Under the Curve (ROC AUC) 1 -Specificity

Losses and Risks § Errors on a different class prediction has different costs What does it cost to mistakenly assign an instance of one class to another? § Normalized Expected Cost § Probability Cost Function Cost matrix Cij Class 1 Class 2 … Class. K Class 1 0 C 12 … C 1 K Class 2 C 21 0 … C 2 K … … … Class. K CK 1 CK 2 … 0 Asymmetric matrix

Cost Curve Normalized expected cost Worse classifier NTP Reject All Classifier Accept All Classifier Actual Classifier NFP Ideal Classifier Probability Cost function

Conclusion § Data mining extracts useful information from datasets § Clustering: Unsupervised Information about the data § Classification: Supervised Build models in order to predict outcome of future observations

Multi-Linear Regression y=ax+b Sum of Squared Errors (SSE) b a