An Adaptive Nearest Neighbor Classification Algorithm for Data
An Adaptive Nearest Neighbor Classification Algorithm for Data Streams Yan-Nei Law & Carlo Zaniolo University of California, Los Angeles PKDD, Porto, 2005 1
Outline n n Related Work ANNCAD Properties of ANNCAD Conclusion 2
Classifying Data Streams n Problem Statement: We seek an algorithm for classifying data streams with numerical attributes--will work for totally ordered domains too. n Desiderata: q Fast update speed for newly arriving records. q Only require single pass of data. q Incremental algorithms are needed. q Coping with concept changes. n Classical mining algorithms were not designed for data streams and need to replaced or modified. 3
Classifying Data Streams: Related Work n Hoeffding trees: q q q n VFDT and CVFDT: build decision tree incrementally. Require a large amount of examples to obtain a fair performance classifier. Unsatisfied performance when training set is small. Ensemble: q q q Combine base models by voting technique. Suitable for coping with concept drift. Fail to provide a simple model and understanding of the problem. 4
State of the Art: Nearest. Neighborhood Classifiers n Pros and cons: q q q n +: Strong intuitive appeal and simple to implement -: Fail to provide simple models/rules -: Expensive Computations ANN: Approximate Nearest Neighborhood with error guarantee 1+ε: q q q Idea: pre-processing the data by devising a data structure (e. g. ring-cover tree) to speed up the searchings. Designed for stored data only. Time for update the pre-processing step depends on size of data set which may be infinite. 5
Our Algorithm: ANNCAD Adaptive NN Classification Algorithm for Data Streams n Model building: q q q n Pre-assign classes to obtain an approximate result and provide simple models/rules. Decompose the feature space to make classification decisions. Akin to wavelets. Classification: q q Find NN for classification adaptively. progressively expand the searching of nearby area of a test point (star). 6
Quantize Feature Space and Compute Multi-resolution Coefficients (8+9+10+0)/4 A set of 100 twoclass training points Quantize Feature Space and record information into data arrays Multi-resolution representation of a two-class data set. 7
Building a Classifier B=6. 75; R=0. 6 Blue Label each block with its majority class B=2; R=4. 25 M(ix) B=3; R=3. 25 M(ix) Label block only if |C 1 st|-|C 2 nd| > 80% Hierarchical structure of ANNCAD Classifier 8
Decision Algorithm on the ANNCAD Hierarchy Classifiedblock Label. Unclassified class. III block, go to next level. The combined classifier over multiple levels Block with tag “M”, go back to prev. level. Compute the distance between the test point and the center of every nonempty neighboring block. 9
Incremental Update 0 8 8 0 10 9 0 0 10 2 1 0 0 0 1 6. 75 2 3. 0625 3 3 0. 25 0. 5 New training point 10
Concept Drift: Adaptation by Exponential Forgetting n Data Array , Factor 0 1: new old n n n No effect if no concept changes Adapt quickly (exponentially) if concept changes No extra memory needed (sliding window required. ) 11
Grid Position and Resolution n n Problem: Neighborhood decision strongly depends on grid position Solution: Build several classifiers by shifting grid position by 1/n. Then combine the results by voting. Example: 4 different grids for building 4 classifiers. n Thm. x: test point, nd classifiers, b(x): Blocks containing x, then: z b(x), y b(x): dist(x, y)<(1+1/n-1)*dist(x, z). q In practice, only 2 -3 classifiers can achieve a good result. 12
Properties of ANNCAD n n Compact support: locality property allows fast update Dealing with noise: can set a threshold for classification decision Multi-resolution: to control the fineness of the result, or optimize the system resources. Low complexity (gd = total number of cells) q Building classifier: O(min(N, gd)) q Testing: O(log 2(g)+2 d). q Updating: log 2(g)+1. 13
n n Experiments Synthetic Data q 3 -d unit cube: q Class distribution: class 0 inside sphere with radius 0. 5 class 1 outside q 3000 training examples q 1000 test examples (a) different initial resolutions. Exact ANN: q q Expand the searching area by double the radius until reaching some training point. Classify the test point with the majority class. (b) different # ensembles. 14
Experiments (Cont’) n Real Data 1 -- Letter Recognition q q q q Objective: identify a pixel displays as one of the 26 letter. 16 numerical attributes to describe its pixel displays. 15, 000 training examples 5, 000 test examples Add 5 % noise by randomly assign class. Grid size: 16 units #Classifiers: 2 15
ANNCAD Vs VFDT n Real Data 2 – Forest Cover Type q q q Objective: predict forest cover type. 10 numerical attributes. 12, 000 training examples 9, 000 test examples Grid size: 32 unit #Classifiers: 2 16
Concept Shift: ANNCAD vs CVFDT n Real Data 3 – Adult q q q Objective: determine a person with salary>50 K Concept Shift Simulation: Group by races = 0. 98 Grid Size: 64 #Classifier: 2 17
Conclusion and Future Work n ANNCAD q q q n an incremental classification algorithm to find adaptive NN Suitable for mining data streams: fast update speed Exponential forgetting for concept shift/drift. Future Work: Detect concept shift/drift by changes in class label of blocks. 18
THANK YOU! 19
- Slides: 19
