2806 Neural Computation Multilayer neural networks Lecture 4

  • Slides: 37
Download presentation
2806 Neural Computation Multilayer neural networks Lecture 4 2005 Ari Visa

2806 Neural Computation Multilayer neural networks Lecture 4 2005 Ari Visa

Agenda Some historical notes n Multilayer neural networks n Backpropagation n Error surfaces and

Agenda Some historical notes n Multilayer neural networks n Backpropagation n Error surfaces and feature mapping n Speed ups n Conclusions n

Some historical notes Rosenblatt’s perceptron 1958: a single neuron with adjustable synaptic weights and

Some historical notes Rosenblatt’s perceptron 1958: a single neuron with adjustable synaptic weights and bias Perceptron convergence theorem 1962 (Rosenblatt): If patterns used to train the perceptron are drawn from two linearly separable classes, then the algorithm converges and positions the decision surface in the form of hyperplane between two classes. The limitations of networks implementing linear discriminants were well known in the 1950 s and 1960 s.

Some historical notes n A single neuron -> the class of solutions that can

Some historical notes n A single neuron -> the class of solutions that can be obtained is not general enough -> multilayer perceptron n Widrow & Hoff (1960): least-mean square algorithm (LMS) = delta rule: three-layer networks designed by hand (fixed input-to-hidden weights, trained hidden-to-output weights). The development of backpropagation: Kalman filtering (Kalman 1960, Bryson, Denham, Dreyfus 1963, 1969). n

Some historical notes Werbos : Beyond Regression: New Tools for Prediction and Analysis in

Some historical notes Werbos : Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Science, Ph. D. thesis, Harvard University, 1974 n Parker 1982, 1985 n Rumelhart, Hinton, Williams: Learning internal representations by backpropagating errors, Nature 323(99), pp 533 -536, 1986 n

Multilayer Neural Networks n n n n Input layer Hidden layer Output layer Bias

Multilayer Neural Networks n n n n Input layer Hidden layer Output layer Bias unit = neuron netj = wtjx yj = f(netj) f(net) = Sgn(net) zk = f(netk)

Multilayer Neural Networks n n n The XOR problem: (x 1 OR x 2)

Multilayer Neural Networks n n n The XOR problem: (x 1 OR x 2) AND NOT (x 1 AND x 2) y 1 : x 1 + x 2 +0. 5 = 0 > 0 -> y 1 = 1, otherwise – 1 y 2 : x 1 + x 2 -1. 5 = 0 > 0 -> y 2 = 1, otherwise – 1

Multilayer Neural Networks n n Any desired continuous function can be implemented by a

Multilayer Neural Networks n n Any desired continuous function can be implemented by a threelayer network given sufficient number of hidden units, proper nonlinearitiers and weights (Kolmogorov) Feedforward Learning (Demo chapter 11)

Multilayer Neural Networks n Nonlinear multilayer networks n How to set the weights of

Multilayer Neural Networks n Nonlinear multilayer networks n How to set the weights of a three-layer neural network based on training patterns and the desired output?

Backpropagation Algorithm n The LMS algorithm exists for linear systems Training error n Learning

Backpropagation Algorithm n The LMS algorithm exists for linear systems Training error n Learning rule n

Backpropagation Algorithm n Learning rule n Hidden-to-output n Input-to-hidden Note, that wij are initialized

Backpropagation Algorithm n Learning rule n Hidden-to-output n Input-to-hidden Note, that wij are initialized with random values Demo Chapter 11 n n

Backpropagation Algorithm n n n Compare with LMS algoritms 1) Method of Steepest Descent

Backpropagation Algorithm n n n Compare with LMS algoritms 1) Method of Steepest Descent The direction of steepest descent is in direction opposite to the gradient vector g = E(w) w(n+1) = w(n) – g(n) is the stepsize or learning-rate parameter

Backpropagation Algorithm n n n Training set = a set of patterns with known

Backpropagation Algorithm n n n Training set = a set of patterns with known labels Stochastic training = patterns are chosen randomly from the training set Batch training = all patterns are presented to the network before learning takes place On-line protocol = each pattern is presented once and only once, no memory in use Epoch = a single presentation of all patterns in the training set. The number of epochs is an indication of the relative amount of learning.

Backpropagation Algorithm n Stopping criterion

Backpropagation Algorithm n Stopping criterion

Backpropagation Algorithm Learning set Validation set Test set Stopping criterion Learning curve, the average

Backpropagation Algorithm Learning set Validation set Test set Stopping criterion Learning curve, the average error per pattern Cross-validation

Error Surfaces and Feature Mapping n Note, error backpropagation is based on gradient descent

Error Surfaces and Feature Mapping n Note, error backpropagation is based on gradient descent in a criterion function J(w) that is represented

Error Surfaces and Feature Mapping n n The total training error is minimized. It

Error Surfaces and Feature Mapping n n The total training error is minimized. It usually decreases monotonically, even though this is not the case for the error on each individual pattern.

Error Surfaces and Feature Mapping n n Hidden-to-output weights ~ a linear discriminant Input-to-hidden

Error Surfaces and Feature Mapping n n Hidden-to-output weights ~ a linear discriminant Input-to-hidden weights ~ ”matched filter”

Practical Techniques for Improving Backpropagation n n How to improve convergence, performance, and results?

Practical Techniques for Improving Backpropagation n n How to improve convergence, performance, and results? Neuron: Sigmoid function = centered zero and antisymmetric

Practical Techniques for Improving Backpropagation Scaling input variables = the input patterns should be

Practical Techniques for Improving Backpropagation Scaling input variables = the input patterns should be shifted so that the average over the training set of each feature is zero. = the full data set should be scaled to have the same variance in each feature component Note, this standardization can only be done for stochastic and batch learning!

Practical Techniques for Improving Backpropagation n n When the training set is small one

Practical Techniques for Improving Backpropagation n n When the training set is small one can generate surrogate training patterns. In the absence of problem-specific information, the surrogate patterns should be made by adding ddimensional Gaussian noise to true training points, the category label should be left unchanged. If we know about the source of variation among patterns we can manufacture training data. The number of hidden units should be less than the total number of training points n, say roughly n/10.

Practical Techniques for Improving Backpropagation n n We cannot initialize the weights to 0.

Practical Techniques for Improving Backpropagation n n We cannot initialize the weights to 0. Initializing weights = uniform learning -> choose weights randomly from a single distribution Input-to-hidden weights: -1 / d < wij < + 1 / d , where d is the number of input units Hidden-to-output weights: -1 / n. H < wkj < + 1 / n. H , where d is the number of hidden units

Practical Techniques for Improving Backpropagation n Learning Rates Demo Chapter 9 The optimal rate

Practical Techniques for Improving Backpropagation n Learning Rates Demo Chapter 9 The optimal rate

Practical Techniques for Improving Backpropagation n Momentum : allows the network to learn more

Practical Techniques for Improving Backpropagation n Momentum : allows the network to learn more quickly when plateaus in the error surface exist. Demo chapter 12. 0. 9

Practical Techniques for Improving Backpropagation Weight Decay to avoid overfitting = to start with

Practical Techniques for Improving Backpropagation Weight Decay to avoid overfitting = to start with a network with too many weights and decay all the weights during training wnew = wold(1 - ), where 0 < < 1 n The weights that are not needed for reducing the error function become smaller and smaller -> eliminated n

Practical Techniques for Improving Backpropagation n n If we have insufficient training data for

Practical Techniques for Improving Backpropagation n n If we have insufficient training data for the desired classification accuracy. Learning with hints is to add output units for addressing an ancillary problem, one different from but related to the specific classification problem at hand.

Practical Techniques for Improving Backpropagation Stopped Training n Number of hidden layers: typically 2

Practical Techniques for Improving Backpropagation Stopped Training n Number of hidden layers: typically 2 -3. n Criterion Function: cross entropy, Minkowski Error n

Practical Techniques for Improving Backpropagation n n Second-Order Methods to speed up the learning

Practical Techniques for Improving Backpropagation n n Second-Order Methods to speed up the learning rate: Newton’s Method, demo chapter 9 Quickprop Conjugate Gradient Descent requires batch training, demo chapter 9

Practical Techniques for Improving Backpropagation n n 2) Newton’s method The idea is to

Practical Techniques for Improving Backpropagation n n 2) Newton’s method The idea is to minimize the quadratic approximation of the cost function E(w) around the current point w(n). Using a second-order Taylor series expansion of the cost function around the point w(n). Ew(n) g. T(n) w(n) +½ w. T(n) H(n) w(n) g(n) is the m-by-1 gradient vector of the cost function E(w) evaluated at the point w(n). The matrix H(n) is the m-by-m Hessian matrix of E(w) (second derivative), H = ²E(w)

Practical Techniques for Improving Backpropagation n n n H = ²E(w) requires the cost

Practical Techniques for Improving Backpropagation n n n H = ²E(w) requires the cost function E(w) to be twice continuously differentiable with respect to the elements of w. Differentiating Ew(n) g. T(n) w(n) +½ w. T(n) H(n) with respect to w, the change E(n) is minimized when g(n) + H(n) w(n) = 0 -> w(n) = H-1(n)g(n) w(n+1) = w(n) + w(n) w(n+1) = w(n)+H-1(n)g(n) where H-1(n) is the inverse of the Hessian of E(w).

Practical Techniques for Improving Backpropagation Newton’s method converges quickly asymtotically and does not exhibit

Practical Techniques for Improving Backpropagation Newton’s method converges quickly asymtotically and does not exhibit the zigzagging behavior. n Newton’s method requires that the Hessian H(n) has to be a positive definite matrix for all n! n

Practical Techniques for Improving Backpropagation 3) Gauss-Newton Method n It is applicable to a

Practical Techniques for Improving Backpropagation 3) Gauss-Newton Method n It is applicable to a cost function that is expressed as the sum of error squares. n E(w) = ½ i=1 n e²(i), note that all the error terms are calculated on the basis of a weight vector w that is fixed over the entire observation interval 1 i n. n The error signal e(i) is a function of the adjustable weight vector w. Given an operating point w(n), we linearize the dependence of e(i) on w by writing e’(i, w) = e(i) + [ e(i)/ w]Tw=w(n) (w-w(n)), i=1, 2, . . . , n n

Practical Techniques for Improving Backpropagation e’(n, w) = e(n) + J(n)(w-w(n)) where e(n) is

Practical Techniques for Improving Backpropagation e’(n, w) = e(n) + J(n)(w-w(n)) where e(n) is the error vector e(n) = [e(1), e(2), . . . , e(n)]T and J(n) is the n-by-m Jacobian matrix of e(n) (The Jacobian J(n) is the transpose of the m-by-n gradient matrix e(n), where e(n) =[ e(1), e(2), . . . , e(n)]. w(n+1) = arg min w {½ e’(n, w) ²} = ½ e(n) ² +e. T(n)J(n)(w-w(n)) + ½(ww(n))TJT(n)J(n)(w-w(n)) Differentiating the expression with respect w and setting the result equal to zero

Practical Techniques for Improving Backpropagation JT(n)e(n) + JT(n)e(n) (w-w(n)) = 0 w(n+1) = w(n)

Practical Techniques for Improving Backpropagation JT(n)e(n) + JT(n)e(n) (w-w(n)) = 0 w(n+1) = w(n) – (JT(n)J(n))-1 JT(n)e(n) The Gauss-Newton requires only the Jacobian matrix of the error function e(n). For the Gauss-Newton iteration to be computable, the matrix product JT(n)J(n) must be nonsigular. JT(n)J(n) is always nonnegative definite but to ensure that it is nonsingular, the Jacobian J(n) must have row rank n. -> add the diagonal matrix I to the matrix JT(n)J(n), the parameter is a small positive constant.

Practical Techniques for Improving Backpropagation JT(n)J(n)+ I ; positve definite for all n. n

Practical Techniques for Improving Backpropagation JT(n)J(n)+ I ; positve definite for all n. n -> The Gauss-Newton method is implemented in the following form: w(n+1) = w(n) – (JT(n)J(n) + I )-1 JT(n)e(n) n This is the solution to the modified cost function: n E(w) = ½{ w-w(0) ²+ i=1 n e²(i)} n where w(0) is the initial value of w. n

Regularization, Complexity Adjustment and Pruning n n n If we have too many weight

Regularization, Complexity Adjustment and Pruning n n n If we have too many weight and train too long > overfitting Wald Statistic: we can estimate the importance of a parameter in a model The Optimal Brain Damage, The Optimal Brain Surgeon: eliminate the parameter having the least importance

Summary n n Multilayer nonlinear neural networks trained by gradient descent methods such as

Summary n n Multilayer nonlinear neural networks trained by gradient descent methods such as backpropagation perform a maximum-likelihood estimation of the weight values in the model defined by the network topology. f(net) at the hidden units allows the networks to form an arbitary decision boundary, so long as there are sufficiently many hidden units.