Universit di MilanoBicocca Laurea Magistrale in Informatica Corso
Università di Milano-Bicocca Laurea Magistrale in Informatica Corso di APPRENDIMENTO E APPROSSIMAZIONE Lezione 8 - Instance based learning Prof. Giancarlo Mauri
Instance-based Learning n Key idea: n n Just store all training examples <xi, f(xi)> Classify a query instance by retrieving a set of “similar” instances Advantages n Training is very fast - just storing examples n Learn complex target functions n Don’t lose information Disadvantages n n Slow at query time - need for efficient indexing of training examples Easily fooled by irrelevant attributes
Instance-based Learning n Main approaches: n k-Nearest Neighbor n Locally weighted regression n Radial basis functions n Case-based reasoning n Lazy and eager learning
K-Nearest Neighbor Learning n Instances: points in Rn n Euclidean distance to measure similarity n Target function f: Rn V n The algorithm: n Given query instance xq ü first locate nearest training example xn, then estimate f(xq) = f(xn) (for k=1) or ü take vote among its k nearest nbrs (k≠ 1 and f discrete-valued) ü take mean of f values of k nearest nbrs (k≠ 1 and f real-valued)
K-Nearest Neighbor Learning n When to Consider Nearest Neighbor n Instances map to points in Rn n Less than 20 attributes per instance n Lots of training data
An example n Boolean target function in 2 D n x classified + for k=1 and - for k=5 (left diagram) n Right diagram (Voronoi diagram) shows the decision surface induced by 1 -nbr for a given set of training examples (black dots) - - x + - - + + +
Voronoi Diagram query point qf nearest neighbor qi
3 -Nearest Neighbors query point qf 3 nearest neighbors 2 x, 1 o
7 -Nearest Neighbors query point qf 7 nearest neighbors 3 x, 4 o
Nearest Neighbor (continuous) 1 -nearest neighbor
Nearest Neighbor (continuous) 3 -nearest neighbor
Nearest Neighbor (continuous) 5 -nearest neighbor
Behavior in the Limit n Let p(x) be the probability that instance x will be labeled 1 (positive) versus 0 (negative) n Nearest neighbor ü As number of training examples ∞ , approaches Gibbs algorithm: with probability p(x) predict 1, else 0 n K-Nearest neighbor ü As number of training examples ∞ and k gets large, approaches Bayes optimal: if p(x)>. 5 then predict 1, else 0 n Note: Gibbs has at most twice the expected error of Bayes optimal
Distance-Weighted k-NN n Might want weight nearer neighbors more heavily… where with d(xq, xi) distance between xq and xi (if xq= xi, i. e. d(xq, xi) = 0, then f(xq) = f(xi)) n Note: n now it makes sense to use all training examples instead of just k (Shepard’s method, global, slow)
Curse of Dimensionality n Imagine instances described by 20 attributes, but only 2 are relevant to target function nearest nbr is easily misled when high-dimensional X n One approach: n n n Stretch jth axis by weight zj, where z 1, …, zn chosen to minimize prediction error Use cross-validation to automatically choose weights z 1, …, zn Note setting zj to zero eliminates this dimension altogether
Locally weighted Regression n Note that k. NN forms local approximation to f for each query point xq : why not form an explicit approximation f’(x) for region surrounding xq ? n Fit linear function to k nearest neighbors n Fit quadratic… n Produces “piecewise approximation” to f n Several choices of error to minimize: n n Squared error over k nearest neighbors E 1(xq) = (∑x NNs of xq(f(x)-f’(x))2)/2 Distance-weighted squared error over all nbrs E 1(xq) = (∑x D(f(x)-f’(x))2 K(d(xq, x))/2
Radial Basis Function Networks n Global approximation to target function, in terms of linear combination of local approximations n Used, e. g. , for image classification n A different kind of neural network n Closely related to distance-weighted regression, but “eager” instead of “lazy”
Radial Basis Function Networks n Where ai(x) are the attributes describing instance x, and One common choice for is
Training RBF Networks Q 1: what xu to use for each kernel function Ku(d(xu, x)) n Scatter uniformly throughout instance space n Or use training instances (reflects instance distribution) Q 2: how to train weights (assume here Gaussian Ku) n First choose variance (and perhaps mean) for each Ku - e. g. , use EM n Then hold Ku fixed, and train linear output layer - efficient methods to fit linear function
Radial Basis Function Network n n Global approximation to target function in terms of linear combination of local approximations Used, e. g. for image classification Similar to back-propagation neural network but activation function is Gaussian rather than sigmoid Closely related to distance-weighted regression but ”eager” instead of ”lazy”
Radial Basis Function Network output f(x) wn linear parameters Kernel functions Kn(d(xn, x))= exp(-1/2 d(xn, x)2/ 2) xi input layer f(x)=w 0+ n=1 k wn Kn(d(xn, x))
Training Radial Basis Function Networks n n How to choose the center xn for each Kernel function Kn? n scatter uniformly across instance space n use distribution of training instances (clustering) How to train the weights? n Choose mean xn and variance n for each Kn nonlinear optimization or EM n Hold Kn fixed and use local linear regression to compute the optimal weights wn
Radial Basis Network Example K 1(d(x 1, x))= exp(-1/2 d(x 1, x)2/ 2) w 1 x+ w 0 f^(x) = K 1 (w 1 x+ w 0) + K 2 (w 3 x + w 2)
Case-based reasoning Can apply instance-based learning even when X ≠ need different “distance” metric Case-based reasoning is instance-based learning applied to instances with symbolic logic descriptions ((user-complaint error 53 -on-shutdown) (cpu-model Power. PC) (operating-system Windows) (network-connection PCIA) (memory 48 meg) (installed-applications Excel Netscape Virus. Scan) (disk 1 gig) (likely-cause? ? ? ))
Case-based reasoning in CADET: 75 stored examples of mechanical devices n each training example: < qualitative function, mechanical structure> n new query: desired function n target value: mechanical structure for this function Distance metric: match qualitative function descriptions
Case-based Reasoning in CADET
Case-based Reasoning in CADET n n n Instances represented by rich structural descriptions Multiple cases retrieved (and combined) to form solution to new problem Tight coupling between case retrieval and problem solving Bottom line: n n Simple matching of cases useful for tasks such as answering help -desk queries Area of ongoing research
Lazy and Eager learning Lazy: wait for query before generalizing n k-NEAREST NEIGHBOR, Case-based reasoning Eager: generalize before seein query n Radial basis function networks, ID 3, Backpropagation, Naive. Bayes, … Does it matter? n Eager learner must create global approximation n Lazy learner can create many local approximations n If they use same H, lazy can represent more complex fns (e. g. , consider H = linear functions)
Machine Learning 2 D 5362 Instance Based Learning
Distance Weighted k-NN Give more weight to neighbors closer to the query point f^(xq) = i=1 k wi f(xi) / i=1 k wi where wi=K(d(xq, xi)) and d(xq, xi) is the distance between xq and xi Instead of only k-nearest neighbors use all training examples (Shepard’s method)
Distance Weighted Average n Weighting the data: f^(xq) = i f(xi) K(d(xi, xq))/ i K(d(xi, xq)) Relevance of a data point (xi, f(xi)) is measured by calculating the distance d(xi, xq) between the query xq and the input vector xi n Weighting the error criterion: E(xq) = i (f^(xq)-f(xi))2 K(d(xi, xq)) the best estimate f^(xq) will minimize the cost E(q), therefore E(q)/ f^(xq)=0
Kernel Functions
Distance Weighted NN K(d(xq, xi)) = 1/ d(xq, xi)2
Distance Weighted NN K(d(xq, xi)) = 1/(d 0+d(xq, xi))2
Distance Weighted NN K(d(xq, xi)) = exp(-(d(xq, xi)/ 0)2)
Linear Global Models n The model is linear in the parameters wk, which can be estimated using a least squares algorithm n f^(xi) = k=1 D k xki or F(x) = X Where xi=(x 1, …, x. D)i, i=1. . N, with D the input dimension and N the number of data points. Estimate the wk by minimizing the error criterion n E= i=1 N (f^(xi) – yi)2 n (XTX) = XT F(X) n = (XT X)-1 XT F(X) n k= m=1 D n=1 N ( l=1 D x. Tkl xlm)-1 x. Tmn f(xn)
Linear Regression Example
Linear Local Models n n Estimate the parameters k such that they locally (near the query point xq) match the training data either by weighting the data: wi=K(d(xi, xq))1/2 and transforming zi=wi xi vi=wi yi n or by weighting the error criterion: E= i=1 N (xi. T – yi)2 K(d(xi, xq)) still linear in with LSQ solution = ((WX)T WX)-1 (WX)T WF(X)
Linear Local Model Example Kernel K(x, xq) Local linear model: f^(x)=b 1 x+b 0 f^(xq)=0. 266 query point Xq=0. 35
Linear Local Model Example
Design Issues in Local Regression n Local model order (constant, linear, quadratic) n Distance function d feature scaling: d(x, q)=( j=1 d mj(xj-qj)2)1/2 irrelevant dimensions mj=0 n kernel function K n smoothing parameter bandwidth h in K(d(x, q)/h) n h=|m| global bandwidth n h= distance to k-th nearest neighbor point n h=h(q) depending on query point n h=hi depending on stored data points See paper by Atkeson [1996] ”Locally Weighted Learning”
Local Linear Models
Local Linear Model Tree (LOLIMOT) • incremental tree construction algorithm • partitions input space by axis-orthogonal splits • adds one local linear model per iteration 1. start with an initial model (e. g. single LLM) 2. identify LLM with worst model error Ei 3. check all divisions : split worst LLM hyper-rectangle in halves along each possible dimension 4. find best (smallest error) out of possible divisions 5. add new validity function and LLM 6. repeat from step 2. until termination criteria is met
LOLIMOT Initial global linear model Split along x 1 or x 2 Pick split that minimizes model error (residual)
LOLIMOT Example
LOLIMOT Example
Lazy and Eager Learning n Lazy: wait for query before generalizing n n k-nearest neighbors, weighted linear regression Eager: generalize before seeing query n Radial basis function networks, decision trees, backpropagation, LOLIMOT n Eager learner must create global approximation n Lazy learner can create local approximations n If they use the same hypothesis space, lazy can represent more complex functions (H=linear functions)
- Slides: 47