Classification by Machine Learning Approaches Michael J Kerner

Classification by Machine Learning Approaches Michael J. Kerner – kerner@cbs. dtu. dk Center for Biological Sequence Analysis Technical University of Denmark

Outline • Introduction to Machine Learning • Datasets, Features • Feature Selection • Machine Learning Approaches (Classifiers) • Model Evaluation and Interpretation • Examples, Exercise

Machine Learning – Data Driven Prediction To Learn: “to gain knowledge or understanding of or skill in by study, instruction, or experience” (Merriam Webster English Dictionary, 2005) Machine Learning: Learning theory automatically from the data, through a process of inference, model fitting, or learning from examples: Automated extraction of useful information from a body of data by building good probabilistic models. Ideally suited for areas with lots of data in the absence of a general theory.

Why do we need Machine Learning? • Some tasks cannot be defined well, except by examples (e. g. recognition of faces or people). • Large amounts of data may have hidden relationships and correlations. Only automated approaches may be able to detect these. • The amount of knowledge about a certain problem / task may be too large for explicit encoding by humans (e. g. in medical diagnostics) • Environments change over time, and new knowledge is constantly being discovered. A continuous redesign of the systems “by hand” may be difficult.

The Machine Learning Approach Input Data ML Classifier e. g. Gene Expression Profiles, … Machine Learning Prediction: Yes / No

Machine Learning • Learning Task: – What do we want to learn or predict? • Data and assumptions: – What data do we have available? – What is their quality? – What can we assume about the given problem? • Representation: – What is a suitable representation of the examples to be classified? • Method and Estimation: – Are there possible hypotheses? – Can we adjust our predictions based on the given results? • Evaluation: – How well does the method perform? – Might another approach/model perform better?

Learning Tasks • Classification: – Prediction of an item class. • Forecasting: – Prediction of a parameter value. • Characterization: – Find hypotheses that describe groups of items. • Clustering: – Partitioning of the (unassigned) data set into clusters with common properties. (Unsupervised learning)

Emergence of Large Datasets Dataset examples: • • Image processing Spam email detection Text mining DNA micro-array data Protein function Protein localization Protein-protein interaction …

Dataset Examples Edible or poisonous ?

Dataset Examples

m. RNA Splicing

m. RNA Splice Site Prediction

Protein Function Prediction: Prot. Fun • Predict as many biologically relevant features as we can from the sequence • Train artificial neural networks for each category • Assign a probability for each category from the NN outputs

####### Prot. Fun 2. 2 predictions #### >KCNA 1_HUMAN # Functional category Prob Odds Amino_acid_biosynthesis 0. 042 1. 893 Biosynthesis_of_cofactors 0. 119 1. 654 Cell_envelope 0. 031 0. 507 Cellular_processes 0. 027 0. 373 Central_intermediary_metabolism 0. 046 0. 731 Energy_metabolism 0. 036 0. 395 Fatty_acid_metabolism 0. 019 1. 485 Purines_and_pyrimidines 0. 214 0. 879 Regulatory_functions 0. 013 0. 083 Replication_and_transcription 0. 019 0. 073 Translation 0. 129 2. 925 Transport_and_binding =>0. 717 1. 748 # Enzyme/nonenzyme Prob Odds Enzyme 0. 231 0. 807 Nonenzyme =>0. 769 1. 078 # Enzyme class Prob Odds Oxidoreductase (EC 1. -. -. -) 0. 040 0. 193 Transferase (EC 2. -. -. -) 0. 056 0. 163 Hydrolase (EC 3. -. -. -) 0. 062 0. 195 Lyase (EC 4. -. -. -) 0. 020 0. 430 Isomerase (EC 5. -. -. -) 0. 010 0. 321 Ligase (EC 6. -. -. -) 0. 017 0. 326 # Gene Ontology category Prob Odds Signal_transducer 0. 061 0. 284 Receptor 0. 055 0. 323 Hormone 0. 001 0. 206 Structural_protein 0. 002 0. 086 Transporter 0. 469 4. 299 Ion_channel 0. 207 3. 633 Voltage-gated_ion_channel =>0. 280 12. 736 Cation_channel 0. 348 7. 560 Transcription 0. 163 1. 270 Transcription_regulation 0. 166 1. 331 Stress_response 0. 011 0. 125 Immune_response 0. 031 0. 370 Growth_factor 0. 005 0. 372 Metal_ion_transport 0. 159 0. 345

Emergence of Large Datasets Complexity of datasets: • Many instances (examples) • Instances with multiple features (properties / characteristics) • Dependencies between the features (correlations)

Data Preprocessing Instance selection: – Remove identical / inconsistent / incomplete instances (e. g. reduction of homologous genes, removal of wrongly annotated genes) Feature transformation / selection: – Projection techniques (e. g. principal components analysis) – Compression techniques (e. g. minimum description length) – Feature selection techniques

Benefits of Feature Selection • Attain good and often even better classification performance using a small subset of features – Less noise in the data • Provide more cost-effective classifiers – Less features to take into account Ø smaller datasets Ø faster classifiers • Identification of (biologically) relevant features for the given problem

Feature Selection Filter approach Wrapper approach All Features Selection Criterion Feature Subset Selection Optimal Features Learning Algorithm All Features Feature Subset Search Algorithm Feature Subset Selection Selected Features Learning Algorithm Evaluation Optimal Features Learning Algorithm

Filter Approach • Independent of the classification model • A relevance measure for each feature is calculated • Features with a value lower than a selected threshold t will be removed Example: Feature-class entropy • Measures the “uncertainty” about the class when observing feature i f 1 f 2 f 3 f 4 class 1 0 1 1 1 1 0 0 0 1 1 0 1 0 0 1 0 1 1 1 0 0 1 1 0 1 0

Wrapper approach • Specific to a classification algorithm • The search for a good feature subset is guided by a search algorithm • The algorithm uses the evaluation of the classifier as a guide to find good feature subsets • Search algorithm examples: sequential forward or backward search, genetic algorithms Sequential backward elimination – Starts with the set of all features – Iteratively discards the feature whose removal results in the best classification performance

Wrapper approach Full feature set : f 1, f 2, f 3, f 4 0. 7 f 3, f 4 f 1, f 3, f 4 0. 85 0. 2 f 1, f 4 f 3 0. 7 f 1, f 2, f 4 0. 1 f 1, f 3 0. 8 f 1, f 2, f 3 0. 75

Classification Methods - Decision trees Hidden Markov Models (HMMs) Support vector machines Artificial Neural Networks Bayesian methods …

Decision Trees • Simple, practical and easy to interpret • Given a set of instances (with a set of features), a tree is constructed with internal nodes as the features and the leaves as the classes

Example Dataset: Shall we play golf? Instance Attributes / Features Class day outlook temperature humidity windy Play Golf ? 1 sunny hot high FALSE no 2 sunny hot high TRUE no 3 overcast hot high FALSE yes 4 rainy mild high FALSE yes 5 rainy cool normal FALSE yes 6 rainy cool normal TRUE no 7 overcast cool normal TRUE yes 8 sunny mild high FALSE no 9 sunny cool normal FALSE yes 10 rainy mild normal FALSE yes 11 sunny mild normal TRUE yes 12 overcast mild high TRUE yes 13 overcast hot normal FALSE yes 14 rainy mild high TRUE no today sunny cool high TRUE ?

Example: Shall we play golf today? WEKA data file (arff format) : @relation weather. symbolic @attribute outlook {sunny, overcast, rainy} @attribute temperature {hot, mild, cool} @attribute humidity {high, normal} @attribute windy {TRUE, FALSE} @attribute play {yes, no} Instance @data sunny, hot, high, FALSE, no sunny, hot, high, TRUE, no overcast, hot, high, FALSE, yes rainy, mild, high, FALSE, yes rainy, cool, normal, TRUE, no overcast, cool, normal, TRUE, yes sunny, mild, high, FALSE, no sunny, cool, normal, FALSE, yes rainy, mild, normal, FALSE, yes sunny, mild, normal, TRUE, yes overcast, mild, high, TRUE, yes overcast, hot, normal, FALSE, yes rainy, mild, high, TRUE, no Independent features (attributes) Class Day Outlook Temperature Humidity Windy Play Golf? 1 sunny hot high FALSE no 2 sunny hot high TRUE no 3 overcast hot high FALSE yes 4 rainy mild high FALSE yes 5 rainy cool normal FALSE yes 6 rainy cool normal TRUE no 7 overcast cool normal TRUE yes 8 sunny mild high FALSE no 9 sunny cool normal FALSE yes 10 rainy mild normal FALSE yes 11 sunny mild normal TRUE yes 12 overcast mild high TRUE yes 13 overcast hot normal FALSE yes 14 rainy mild high TRUE no

NO YES False True normal high mild cool hot rainy overcast sunny Feature compositions NO YES

Decision Trees J 48 pruned tree ---------outlook = sunny | humidity = high: no (3. 0) | humidity = normal: yes (2. 0) outlook = overcast: yes (4. 0) outlook = rainy | windy = TRUE: no (2. 0) | windy = FALSE: yes (3. 0) Number of Leaves : Size of the tree : 5 8 Attributes / Features Attribute Values Classes

Artificial Neural Networks (ANNs) Artificial Neuron Neural Network

Overfitting: A classifier that performs well on the training examples, but poorly on new examples. Training and testing on the same data will generally produce a good classifier (for this dataset) with high overfitting. To avoid overfitting: • Use separate training and testing data • Use cross-validation • Use the simplest model possible

Performance Evaluation Cross-Validation (10 fold) Data (9/10) Training Set Test Set (1/10) ML Classifier Performance Evaluation 10 x

Performance Evaluation Predicted Label Known positive Label negative positive negative TP FP FN TN Confusion Matrix TP True Positives TN True Negatives FP False Positives FN False Negatives

Performance Evaluation • Precision (PPV) TP / (TP + FP) – Percentage of correct positive predictions • Recall / Sensitivity TP / (TP + FN) – Percentage of positively labeled instances, also predicted as positive • Specificity TN / (TN + FP) – Percentage of negatively labeled instances, also predicted as negative • Accuracy (TP + TN) / (TP + TN + FP + FN) – Percentage of correct predictions • Correlation Coefficient (TP * TN – FP * FN) (TP+FP)*(FP+TN)*(TN+FN)*(FN+TP) -1 ≤ cc ≤ 1 cc = 1 : no FP or FN cc = 0 : random cc = -1: only FP and FN

True Positive Rate, Sensitivity TP / (TP + FN) ROC – Receiver Operating Characteristic ( FP / (FP + TN) ) False Positive Rate, (1 - Specificity)

Sensitivity ROC – Receiver Operating Characteristic 1 - Specificity

Case Study - Splice Site Prediction

Case Study - Splice Site Prediction Splice site prediction: Correctly identify the borders of introns and exons in genes (splice sites) • Important for gene prediction • Split up into 2 tasks: – Donor prediction (exon -> intron) – Acceptor prediction (intron -> exon)

Case Study - Splice Site Prediction • Splice sites are characterized by a conserved dinucleotide in the intron part of the sequence – Donor sites : – Acceptor sites : • Classification problem: – Distinguish between true GT, AG and false GT, AG.

Case Study - Splice Site Prediction Features: • Position dependent features 1 2 3 17 28 atcgatcagtatcgat GT ctgagctatgag e. g. an A on position 1, C on position 17, …. • Position independent features atcgatcagtatcgat GT ctgagctatgag e. g. subsequence “TCG” occurs, “GAG” occurs, …

Original Data – Human Acceptor Splice Sites >HUMGLUT 4 B_3535 GGGCCCCTAGCGGAAAAAAATCATGGTTCCATGTGACATGCTGTGTCTTTGTGTCTGCCTGTTCAGGATGGGGAACCCCCTCAGCA >HUMGLUT 4 B_3763 GAGGACAGGTGTCTCGGGGGTGGTGGAAAGGGGACGGTCTGCAGGAAATCTGTCCTCTGCTGTCCCCCAGGTGATTGAACAGAGCTACAA >HUMGLUT 4 B_4028 TGGGGGAAACAGGAAGGGAGCCACTGCTGGGTGCCCTCACAGCCTCACTCTGCCTGCCAGGAAAAGGGCCATGCTGGTCA >HUMGLUT 4 B_4276 TGGGCTTTCAGATGGGAATGGACACCTGCCCTCAGCCCTCTCTTCTTCCCTCGCCCAGGGCTGACATCAGGGCTGGTGCCCATGTACGTG >HUMGLUT 4 B_4507 ATATGGTGGGCTTCCAAGGTAAGGCAGAAGGGCTGAGTGACCTGCCTTCTTTCCCAACCTTCTCCCACAGGTGCTGGGCTTGGAGTCCCT >HUMGLUT 4 B_4775 GCCTCCGCCTCATCTTGCTAGCACCTGGCTTCCTCTCAGGTCCCCTCAGGCCTGACCTTCTCCAGGTCTGAAGCGCCTGACAGGC >HUMGLUT 4 B_5125 CCAGCCTGTTGTGGCTGGAGTAGAGGAAGGGGCATTCCTGCCATCACTTCTTCTTCTCCCCCACCTCTAGGTTTTCTATTATTCGACCAG >HUMGLUT 4 B_5378 CCTCACCCACGCGGCCCCTCCTACTTCCCGTGCCCAAAAGGCTGGGGTCAAGCTCCGACTCTCCCCGCAGGTGTTGTTGGTGGAGCGGGC >HUMGLUT 4 B_5995 CTGAGTTGAGGGCAAGGGAAGATCAGAAAGGCCTCAACTGGATTCTCCACCCTGTCTGGCCCCTAGGAGCGAGTTCCAGCCATGAG >HUMGLUT 4 B_6716 CTGGTTGCCTGAAACTACCCCTTCCCCACCTCACTCCGTCAACACCTCTTTCTCCACCTGTCCCAGGAGGCTATGGGGCCCTACGT >HSRPS 6 G_1493 CTTTGTAGATGGCTCTACAATTACCTGTATAGTTTCGTAAACTATTTCCCCCCTTTTAATCCTTAGCTGAACATCTCCTTCCCAGC [. . . ]

Arff Data File - WEKA @RELATION splice-train @ATTRIBUTE @ATTRIBUTE [. . . ] @ATTRIBUTE @ATTRIBUTE -68_A -68_T -68_C -68_G -67_A -67_T -67_C -67_G {0, 1} {0, 1} 20_A {0, 1} 20_T {0, 1} 20_C {0, 1} 20_G {0, 1} class {true, false} @DATA 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, [. . . ] 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, [. . . ] , 1, 0, 0, 0, true , 0, 0, 0, 1, true [. . . ] , 0, 1, 0, 0, true , 0, 0, 1, 0, true , 0, 0, 0, 1, true , 0, 0, 1, 0, true The original sequence files in FASTA format have been converted to represent the four DNA bases in a binary fashion A: 1 0 0 0 T: 0 1 0 0 C: 0 0 1 0 G: 0 0 0 1

Case Study - Splice Site Prediction • Local context of 88 nucleotides around the splice site • 88 position dependent features • A=1000, T=0100, C=0010, G=0001 Ø 352 binary features • Reduce the dataset to contain fewer but relevant features

352 Binary features

15 Binary features

+ 4 + 3 + 2 + 1 - 2 - 3 + 1 - 2 - 1 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 - 11 - 12 - 13 - 14 - 15 - 16 - 17 - 18 Case Study – Splice Site Sequence Logos Acceptor Sites: Donor Sites:

Exercise: • Building a prediction tool for human m. RNA splice sites • Feature selection for classification of splice sites • Tool: The WEKA machine learning toolkit. • Go to http: //www. cbs. dtu. dk/~kerner/Gene. Disc_Course_2007_MJK/ and follow the instructions

Acknowledgements Slides and Exercises Adapted from and inspired by: Søren Brunak David Gilbert, Aik Choon Tan Yvan Saeys