Machine Learning Lecture 6 KNearest Neighbor Classifier G

Machine Learning Lecture 6 K-Nearest Neighbor Classifier G 53 MLE Machine Learning Dr Guoping Qiu 1

Objects, Feature Vectors, Points x 1 X(15) X(1) X(3) X(25) X(9) X(12) X(7) X(16) X(8) 3 4 5 6 7 10 9 11 12 X(13) X(6) 13 14 X(10) X(4) X(11) X(14) 2 1 15 8 16 Elliptical blobs (objects) x 2 2

Nearest Neighbours x 1 X(j)=(x 1(j), x 2(j), …, xn(j)) X(i)=(x 1(i), x 2(i), …, xn(i)) x 2 G 53 MLE Machine Learning Dr Guoping Qiu 3

Nearest Neighbour Algorithm • Given training data (X(1), D(1)), (X(2), D(2)), …, (X(N), D(N)) • Define a distance metric between points in inputs space. Common measures are: Euclidean Distance G 53 MLE Machine Learning Dr Guoping Qiu 4

K-Nearest Neighbour Model Given test point X • Find the K nearest training inputs to X • Denote these points as (X(1), D(1)), (X(2), D(2)), …, (X(k), D(k)) G 53 MLE Machine Learning Dr Guoping Qiu x 5

K-Nearest Neighbour Model Classification • The class identification of X Y = most common class in set {D(1), D(2), …, D(k)} x x G 53 MLE Machine Learning Dr Guoping Qiu 6

K-Nearest Neighbour Model • Example : Classify whether a customer will respond to a survey question using a 3 -Nearest Neighbor classifier Customer Age Income No. credit cards Response John 35 35 K 3 No Rachel 22 50 K 2 Yes Hannah 63 200 K 1 No Tom 59 170 K 1 No Nellie 25 40 K 4 Yes David 37 50 K 2 ? G 53 MLE Machine Learning Dr Guoping Qiu 7

K-Nearest Neighbour Model • Example : 3 -Nearest Neighbors 15. 16 15 15. 74 G 53 MLE Machine Learning Dr Guoping Qiu 122 152. 23 8

K-Nearest Neighbour Model • Example : 3 -Nearest Neighbors 15. 16 15 15. 74 122 152. 23 Three nearest ones to David are: No, Yes G 53 MLE Machine Learning Dr Guoping Qiu 9

K-Nearest Neighbour Model • Example : 3 -Nearest Neighbors 15. 16 15 Yes 15. 74 122 152. 23 Three nearest ones to David are: No, Yes G 53 MLE Machine Learning Dr Guoping Qiu 10

K-Nearest Neighbour Model • Picking K – Use N fold cross validation – Pick K to minimize the cross validation error – For each of N training example • • – Find its K nearest neighbours Make a classification based on these K neighbours Calculate classification error Output average error over all examples Use the K that gives lowest average error over the N training examples G 53 MLE Machine Learning Dr Guoping Qiu 11

K-Nearest Neighbour Model • Example: For the example we saw earlier, pick the best K from the set {1, 2, 3} to build a K-NN classifier G 53 MLE Machine Learning Dr Guoping Qiu 12

Further Readings 1. T. M. Mitchell, Machine Learning, Mc. Graw-Hill International Edition, 1997 Chapter 8 G 53 MLE Machine Learning Dr Guoping Qiu 13

Tutorial/Exercise Questions 1. K nearest neighbor classifier has to store all training data creating high requirement on storage. Can you think of ways to reduce the storage requirement without affecting the performance? (hint: search the Internet, you will find many approximation methods). G 53 MLE Machine Learning Dr Guoping Qiu 14