Lecture Slides for INTRODUCTION TO Machine Learning 2

  • Slides: 30
Download presentation
Lecture Slides for INTRODUCTION TO Machine Learning 2 nd Edition ETHEM ALPAYDIN © The

Lecture Slides for INTRODUCTION TO Machine Learning 2 nd Edition ETHEM ALPAYDIN © The MIT Press, 2010 alpaydin@boun. edu. tr http: //www. cmpe. boun. edu. tr/~ethem/i 2 ml 2 e

CHAPTER 19: Design and Analysis of Machine Learning Experiments

CHAPTER 19: Design and Analysis of Machine Learning Experiments

Introduction �Questions: �Assessment of the expected error of a learning algorithm: Is the error

Introduction �Questions: �Assessment of the expected error of a learning algorithm: Is the error rate of 1 -NN less than 2%? �Comparing the expected errors of two algorithms: Is k-NN more accurate than MLP ? �Training/validation/test sets �Resampling methods: K-fold cross-validation Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 3

Algorithm Preference �Criteria (Application-dependent): �Misclassification error, or risk (loss functions) �Training time/space complexity �Testing

Algorithm Preference �Criteria (Application-dependent): �Misclassification error, or risk (loss functions) �Training time/space complexity �Testing time/space complexity �Interpretability �Easy programmability �Cost-sensitive learning Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 4

Factors and Response Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2

Factors and Response Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 5

Strategies of Experimentation Response surface design for approximating and maximizing the response function in

Strategies of Experimentation Response surface design for approximating and maximizing the response function in terms of the controllable factors Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 6

Guidelines for ML experiments A. B. C. D. E. F. G. Aim of the

Guidelines for ML experiments A. B. C. D. E. F. G. Aim of the study Selection of the response variable Choice of factors and levels Choice of experimental design Performing the experiment Statistical Analysis of the Data Conclusions and Recommendations Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 7

Resampling and K-Fold Cross-Validation �The need for multiple training/validation sets {Xi, Vi}i: Training/validation sets

Resampling and K-Fold Cross-Validation �The need for multiple training/validation sets {Xi, Vi}i: Training/validation sets of fold i �K-fold cross-validation: Divide X into k, Xi, i=1, . . . , K �Ti share K-2 parts Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 8

5× 2 Cross-Validation � 5 times 2 fold cross-validation (Dietterich, 1998) Lecture Notes for

5× 2 Cross-Validation � 5 times 2 fold cross-validation (Dietterich, 1998) Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 9

Bootstrapping �Draw instances from a dataset with replacement �Prob that we do not pick

Bootstrapping �Draw instances from a dataset with replacement �Prob that we do not pick an instance after N draws that is, only 36. 8% is new! Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 10

Measuring Error �Error rate �Recall = # of errors / # of instances =

Measuring Error �Error rate �Recall = # of errors / # of instances = (FN+FP) / N = # of found positives / # of positives = TP / (TP+FN) = sensitivity = hit rate �Precision = # of found positives / # of found = TP / (TP+FP) �Specificity = TN / (TN+FP) �False alarm rate = FP / (FP+TN) = 1 - Specificity Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 11

ROC Curve Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e

ROC Curve Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 12

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 13

Precision and Recall Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2

Precision and Recall Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 14

Interval Estimation �X = { xt }t where xt ~ N ( μ, σ2)

Interval Estimation �X = { xt }t where xt ~ N ( μ, σ2) �m ~ N ( μ, σ2/N) 100(1 - α) percent confidence interval Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 15

When σ2 is not known: Lecture Notes for E Alpaydın 2010 Introduction to Machine

When σ2 is not known: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 16

Hypothesis Testing �Reject a null hypothesis if not supported by the sample with enough

Hypothesis Testing �Reject a null hypothesis if not supported by the sample with enough confidence �X = { xt }t where xt ~ N ( μ, σ2) H 0: μ = μ 0 vs. H 1: μ ≠ μ 0 Accept H 0 with level of significance α if μ 0 is in the 100(1 - α) confidence interval Two-sided test Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 17

�One-sided test: H 0: μ ≤ μ 0 vs. H 1: μ > μ

�One-sided test: H 0: μ ≤ μ 0 vs. H 1: μ > μ 0 Accept if �Variance unknown: Use t, instead of z Accept H 0: μ = μ 0 if Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 18

Assessing Error: H 0: p ≤ p 0 vs. H 1: p > p

Assessing Error: H 0: p ≤ p 0 vs. H 1: p > p 0 �Single training/validation set: Binomial Test If error prob is p 0, prob that there are e errors or less in N validation trials is Accept if this prob is less than 1 - α N=100, e=20 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 19

Normal Approximation to the Binomial �Number of errors X is approx N with mean

Normal Approximation to the Binomial �Number of errors X is approx N with mean Np 0 and var Np 0(1 -p 0) Accept if this prob for X = e is less than z 1 -α 1 - α Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 20

t Test �Multiple training/validation sets �xti = 1 if instance t misclassified on fold

t Test �Multiple training/validation sets �xti = 1 if instance t misclassified on fold i �Error rate of fold i: �With m and s 2 average and var of pi , we accept p 0 or less error if is less than tα, K-1 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 21

Comparing Classifiers: H 0: μ 0 = μ 1 vs. H 1: μ 0

Comparing Classifiers: H 0: μ 0 = μ 1 vs. H 1: μ 0 ≠ μ 1 �Single training/validation set: Mc. Nemar’s Test �Under H 0, we expect e 01= e 10=(e 01+ e 10)/2 Accept if < X 2α, 1 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 22

K-Fold CV Paired t Test �Use K-fold cv to get K training/validation folds �pi

K-Fold CV Paired t Test �Use K-fold cv to get K training/validation folds �pi 1, pi 2: Errors of classifiers 1 and 2 on fold i �pi = pi 1 – pi 2 : Paired difference on fold i �The null hypothesis is whether pi has mean 0 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 23

5× 2 cv Paired t Test �Use 5× 2 cv to get 2 folds

5× 2 cv Paired t Test �Use 5× 2 cv to get 2 folds of 5 tra/val replications (Dietterich, 1998) �pi(j) : difference btw errors of 1 and 2 on fold j=1, 2 of replication i=1, . . . , 5 Two-sided test: Accept H 0: μ 0 = μ 1 if in (-tα/2, 5, tα/2, 5) One-sided test: Accept H 0: μ 0 ≤ μ 1 if < tα, 5 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 24

5× 2 cv Paired F Test Two-sided test: Accept H 0: μ 0 =

5× 2 cv Paired F Test Two-sided test: Accept H 0: μ 0 = μ 1 if < Fα, 10, 5 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 25

Comparing L>2 Algorithms: Analysis of Variance (Anova) �Errors of L algorithms on K folds

Comparing L>2 Algorithms: Analysis of Variance (Anova) �Errors of L algorithms on K folds �We construct two estimators to σ2. One is valid if H 0 is true, the other is always valid. We reject H 0 if the two estimators disagree. Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 26

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 27

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 28

ANOVA table If ANOVA rejects, we do pairwise posthoc tests Lecture Notes for E

ANOVA table If ANOVA rejects, we do pairwise posthoc tests Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 29

Comparison over Multiple Datasets �Comparing two algorithms: Sign test: Count how many times A

Comparison over Multiple Datasets �Comparing two algorithms: Sign test: Count how many times A beats B over N datasets, and check if this could have been by chance if A and B did have the same error rate �Comparing multiple algorithms Kruskal-Wallis test: Calculate the average rank of all algorithms on N datasets, and check if these could have been by chance if they all had equal error If KW rejects, we do pairwise posthoc tests to find which ones have significant rank difference Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 30

Machine Learning Machine Learning Machine learningstatistical learning artificialMachine Learning Machine Learning Machine learningstatistical learning artificial
What is Machine Learning Machine learning defined MachineWhat is Machine Learning Machine learning defined Machine
Machine Learning Spark machine learning Spark Mllib MachineMachine Learning Spark machine learning Spark Mllib Machine
What is Machine Learning Machine learning defined MachineWhat is Machine Learning Machine learning defined Machine
Machine Learnin Andrew Ng Machine Learning summary MachineMachine Learnin Andrew Ng Machine Learning summary Machine
Machine Learning Lecture 1 Overview of Machine LearningMachine Learning Lecture 1 Overview of Machine Learning
Machine Learning Lecture 1 Contents Why machine learningMachine Learning Lecture 1 Contents Why machine learning
Paiva Slides Paiva Slides Paiva Slides Paiva SlidesPaiva Slides Paiva Slides Paiva Slides Paiva Slides
Paiva Slides Paiva Slides Paiva Slides Paiva SlidesPaiva Slides Paiva Slides Paiva Slides Paiva Slides
Introduction Machine Learning 14022017 Machine Learning How canIntroduction Machine Learning 14022017 Machine Learning How can
Machine Learning Introduction Machine Learning Tom T MitchellMachine Learning Introduction Machine Learning Tom T Mitchell
Machine Learning Introduction CSE 4309 Machine Learning VassilisMachine Learning Introduction CSE 4309 Machine Learning Vassilis
Introduction to Machine Learning What is Machine LearningIntroduction to Machine Learning What is Machine Learning
Machine Learning Introduction CSE 4309 Machine Learning VassilisMachine Learning Introduction CSE 4309 Machine Learning Vassilis
Introduction What is machine learning Machine Learning AndrewIntroduction What is machine learning Machine Learning Andrew
Machine Learning Introduction Why is machine learning importantMachine Learning Introduction Why is machine learning important
Machine Learning Introduction CSE 4309 Machine Learning VassilisMachine Learning Introduction CSE 4309 Machine Learning Vassilis
Machine Learning Introduction Why is machine learning importantMachine Learning Introduction Why is machine learning important
Machine Learning Introduction Why is machine learning importantMachine Learning Introduction Why is machine learning important
Machine Learning Introduction Machine Learning Tom T MitchellMachine Learning Introduction Machine Learning Tom T Mitchell
AZURE MACHINE LEARNING INTRODUCTION AU MACHINE LEARNING AVECAZURE MACHINE LEARNING INTRODUCTION AU MACHINE LEARNING AVEC
Machine Learning Introduction Jeff Howbert Introduction to MachineMachine Learning Introduction Jeff Howbert Introduction to Machine
Introduction Welcome Machine Learning Introduction What is machineIntroduction Welcome Machine Learning Introduction What is machine
Lecture Slides for INTRODUCTION TO Machine Learning ETHEMLecture Slides for INTRODUCTION TO Machine Learning ETHEM