Lecture Slides for INTRODUCTION TO Machine Learning 2

  • Slides: 24
Download presentation
Lecture Slides for INTRODUCTION TO Machine Learning 2 nd Edition ETHEM ALPAYDIN © The

Lecture Slides for INTRODUCTION TO Machine Learning 2 nd Edition ETHEM ALPAYDIN © The MIT Press, 2010 alpaydin@boun. edu. tr http: //www. cmpe. boun. edu. tr/~ethem/i 2 ml 2 e

CHAPTER 10: Linear Discrimination

CHAPTER 10: Linear Discrimination

Likelihood- vs. Discriminant-based Classification �Likelihood-based: Assume a model for p(x|Ci), use Bayes’ rule to

Likelihood- vs. Discriminant-based Classification �Likelihood-based: Assume a model for p(x|Ci), use Bayes’ rule to calculate P(Ci|x) gi(x) = log P(Ci|x) �Discriminant-based: Assume a model for gi(x|Φi); no density estimation �Estimating the boundaries is enough; no need to accurately estimate the densities inside the boundaries Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 3

Linear Discriminant �Linear discriminant: �Advantages: �Simple: O(d) space/computation �Knowledge extraction: Weighted sum of attributes;

Linear Discriminant �Linear discriminant: �Advantages: �Simple: O(d) space/computation �Knowledge extraction: Weighted sum of attributes; positive/negative weights, magnitudes (credit scoring) �Optimal when p(x|Ci) are Gaussian with shared cov matrix; useful when classes are (almost) linearly separable Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 4

Generalized Linear Model �Quadratic discriminant: �Higher-order (product) terms: Map from x to z using

Generalized Linear Model �Quadratic discriminant: �Higher-order (product) terms: Map from x to z using nonlinear basis functions and use a linear discriminant in z-space Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 5

Two Classes Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e

Two Classes Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 6

Geometry Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e ©

Geometry Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 7

Multiple Classes are linearly separable Lecture Notes for E Alpaydın 2010 Introduction to Machine

Multiple Classes are linearly separable Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 8

Pairwise Separation Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e

Pairwise Separation Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 9

From Discriminants to Posteriors When p (x | Ci ) ~ N ( μi

From Discriminants to Posteriors When p (x | Ci ) ~ N ( μi , ∑) Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 10

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 11

Sigmoid (Logistic) Function Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2

Sigmoid (Logistic) Function Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 12

Gradient-Descent �E(w|X) is error with parameters w on sample X w*=arg minw E(w |

Gradient-Descent �E(w|X) is error with parameters w on sample X w*=arg minw E(w | X) �Gradient-descent: Starts from random w and updates w iteratively in the negative direction of gradient Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 13

Gradient-Descent E (wt) E (wt+1) wt η wt+1 Lecture Notes for E Alpaydın 2010

Gradient-Descent E (wt) E (wt+1) wt η wt+1 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 14

Logistic Discrimination �Two classes: Assume log likelihood ratio is linear Lecture Notes for E

Logistic Discrimination �Two classes: Assume log likelihood ratio is linear Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 15

Training: Two Classes Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2

Training: Two Classes Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 16

Training: Gradient-Descent Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e

Training: Gradient-Descent Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 17

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 18

100 10 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e

100 10 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 19

K>2 Classes softmax Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2

K>2 Classes softmax Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 20

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 21

Example Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e ©

Example Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 22

Generalizing the Linear Model � Quadratic: � Sum of basis functions: where φ(x) are

Generalizing the Linear Model � Quadratic: � Sum of basis functions: where φ(x) are basis functions � Hidden units in neural networks (Chapters 11 and 12) � Kernels in SVM (Chapter 13) Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 23

Discrimination by Regression �Classes are NOT mutually exclusive and exhaustive Lecture Notes for E

Discrimination by Regression �Classes are NOT mutually exclusive and exhaustive Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2 e © The MIT Press (V 1. 0) 24

Machine Learning Machine Learning Machine learningstatistical learning artificialMachine Learning Machine Learning Machine learningstatistical learning artificial
What is Machine Learning Machine learning defined MachineWhat is Machine Learning Machine learning defined Machine
Machine Learning Spark machine learning Spark Mllib MachineMachine Learning Spark machine learning Spark Mllib Machine
What is Machine Learning Machine learning defined MachineWhat is Machine Learning Machine learning defined Machine
Machine Learnin Andrew Ng Machine Learning summary MachineMachine Learnin Andrew Ng Machine Learning summary Machine
Machine Learning Lecture 1 Overview of Machine LearningMachine Learning Lecture 1 Overview of Machine Learning
Machine Learning Lecture 1 Contents Why machine learningMachine Learning Lecture 1 Contents Why machine learning
Paiva Slides Paiva Slides Paiva Slides Paiva SlidesPaiva Slides Paiva Slides Paiva Slides Paiva Slides
Paiva Slides Paiva Slides Paiva Slides Paiva SlidesPaiva Slides Paiva Slides Paiva Slides Paiva Slides
Introduction Machine Learning 14022017 Machine Learning How canIntroduction Machine Learning 14022017 Machine Learning How can
Machine Learning Introduction Machine Learning Tom T MitchellMachine Learning Introduction Machine Learning Tom T Mitchell
Machine Learning Introduction CSE 4309 Machine Learning VassilisMachine Learning Introduction CSE 4309 Machine Learning Vassilis
Introduction to Machine Learning What is Machine LearningIntroduction to Machine Learning What is Machine Learning
Machine Learning Introduction CSE 4309 Machine Learning VassilisMachine Learning Introduction CSE 4309 Machine Learning Vassilis
Introduction What is machine learning Machine Learning AndrewIntroduction What is machine learning Machine Learning Andrew
Machine Learning Introduction Why is machine learning importantMachine Learning Introduction Why is machine learning important
Machine Learning Introduction CSE 4309 Machine Learning VassilisMachine Learning Introduction CSE 4309 Machine Learning Vassilis
Machine Learning Introduction Why is machine learning importantMachine Learning Introduction Why is machine learning important
Machine Learning Introduction Why is machine learning importantMachine Learning Introduction Why is machine learning important
Machine Learning Introduction Machine Learning Tom T MitchellMachine Learning Introduction Machine Learning Tom T Mitchell
AZURE MACHINE LEARNING INTRODUCTION AU MACHINE LEARNING AVECAZURE MACHINE LEARNING INTRODUCTION AU MACHINE LEARNING AVEC
Machine Learning Introduction Jeff Howbert Introduction to MachineMachine Learning Introduction Jeff Howbert Introduction to Machine
Introduction Welcome Machine Learning Introduction What is machineIntroduction Welcome Machine Learning Introduction What is machine