2806 Neural Computation Multilayer neural networks Lecture 4
- Slides: 37
2806 Neural Computation Multilayer neural networks Lecture 4 2005 Ari Visa
Agenda Some historical notes n Multilayer neural networks n Backpropagation n Error surfaces and feature mapping n Speed ups n Conclusions n
Some historical notes Rosenblatt’s perceptron 1958: a single neuron with adjustable synaptic weights and bias Perceptron convergence theorem 1962 (Rosenblatt): If patterns used to train the perceptron are drawn from two linearly separable classes, then the algorithm converges and positions the decision surface in the form of hyperplane between two classes. The limitations of networks implementing linear discriminants were well known in the 1950 s and 1960 s.
Some historical notes n A single neuron -> the class of solutions that can be obtained is not general enough -> multilayer perceptron n Widrow & Hoff (1960): least-mean square algorithm (LMS) = delta rule: three-layer networks designed by hand (fixed input-to-hidden weights, trained hidden-to-output weights). The development of backpropagation: Kalman filtering (Kalman 1960, Bryson, Denham, Dreyfus 1963, 1969). n
Some historical notes Werbos : Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Science, Ph. D. thesis, Harvard University, 1974 n Parker 1982, 1985 n Rumelhart, Hinton, Williams: Learning internal representations by backpropagating errors, Nature 323(99), pp 533 -536, 1986 n
Multilayer Neural Networks n n n n Input layer Hidden layer Output layer Bias unit = neuron netj = wtjx yj = f(netj) f(net) = Sgn(net) zk = f(netk)
Multilayer Neural Networks n n n The XOR problem: (x 1 OR x 2) AND NOT (x 1 AND x 2) y 1 : x 1 + x 2 +0. 5 = 0 > 0 -> y 1 = 1, otherwise – 1 y 2 : x 1 + x 2 -1. 5 = 0 > 0 -> y 2 = 1, otherwise – 1
Multilayer Neural Networks n n Any desired continuous function can be implemented by a threelayer network given sufficient number of hidden units, proper nonlinearitiers and weights (Kolmogorov) Feedforward Learning (Demo chapter 11)
Multilayer Neural Networks n Nonlinear multilayer networks n How to set the weights of a three-layer neural network based on training patterns and the desired output?
Backpropagation Algorithm n The LMS algorithm exists for linear systems Training error n Learning rule n
Backpropagation Algorithm n Learning rule n Hidden-to-output n Input-to-hidden Note, that wij are initialized with random values Demo Chapter 11 n n
Backpropagation Algorithm n n n Compare with LMS algoritms 1) Method of Steepest Descent The direction of steepest descent is in direction opposite to the gradient vector g = E(w) w(n+1) = w(n) – g(n) is the stepsize or learning-rate parameter
Backpropagation Algorithm n n n Training set = a set of patterns with known labels Stochastic training = patterns are chosen randomly from the training set Batch training = all patterns are presented to the network before learning takes place On-line protocol = each pattern is presented once and only once, no memory in use Epoch = a single presentation of all patterns in the training set. The number of epochs is an indication of the relative amount of learning.
Backpropagation Algorithm n Stopping criterion
Backpropagation Algorithm Learning set Validation set Test set Stopping criterion Learning curve, the average error per pattern Cross-validation
Error Surfaces and Feature Mapping n Note, error backpropagation is based on gradient descent in a criterion function J(w) that is represented
Error Surfaces and Feature Mapping n n The total training error is minimized. It usually decreases monotonically, even though this is not the case for the error on each individual pattern.
Error Surfaces and Feature Mapping n n Hidden-to-output weights ~ a linear discriminant Input-to-hidden weights ~ ”matched filter”
Practical Techniques for Improving Backpropagation n n How to improve convergence, performance, and results? Neuron: Sigmoid function = centered zero and antisymmetric
Practical Techniques for Improving Backpropagation Scaling input variables = the input patterns should be shifted so that the average over the training set of each feature is zero. = the full data set should be scaled to have the same variance in each feature component Note, this standardization can only be done for stochastic and batch learning!
Practical Techniques for Improving Backpropagation n n When the training set is small one can generate surrogate training patterns. In the absence of problem-specific information, the surrogate patterns should be made by adding ddimensional Gaussian noise to true training points, the category label should be left unchanged. If we know about the source of variation among patterns we can manufacture training data. The number of hidden units should be less than the total number of training points n, say roughly n/10.
Practical Techniques for Improving Backpropagation n n We cannot initialize the weights to 0. Initializing weights = uniform learning -> choose weights randomly from a single distribution Input-to-hidden weights: -1 / d < wij < + 1 / d , where d is the number of input units Hidden-to-output weights: -1 / n. H < wkj < + 1 / n. H , where d is the number of hidden units
Practical Techniques for Improving Backpropagation n Learning Rates Demo Chapter 9 The optimal rate
Practical Techniques for Improving Backpropagation n Momentum : allows the network to learn more quickly when plateaus in the error surface exist. Demo chapter 12. 0. 9
Practical Techniques for Improving Backpropagation Weight Decay to avoid overfitting = to start with a network with too many weights and decay all the weights during training wnew = wold(1 - ), where 0 < < 1 n The weights that are not needed for reducing the error function become smaller and smaller -> eliminated n
Practical Techniques for Improving Backpropagation n n If we have insufficient training data for the desired classification accuracy. Learning with hints is to add output units for addressing an ancillary problem, one different from but related to the specific classification problem at hand.
Practical Techniques for Improving Backpropagation Stopped Training n Number of hidden layers: typically 2 -3. n Criterion Function: cross entropy, Minkowski Error n
Practical Techniques for Improving Backpropagation n n Second-Order Methods to speed up the learning rate: Newton’s Method, demo chapter 9 Quickprop Conjugate Gradient Descent requires batch training, demo chapter 9
Practical Techniques for Improving Backpropagation n n 2) Newton’s method The idea is to minimize the quadratic approximation of the cost function E(w) around the current point w(n). Using a second-order Taylor series expansion of the cost function around the point w(n). Ew(n) g. T(n) w(n) +½ w. T(n) H(n) w(n) g(n) is the m-by-1 gradient vector of the cost function E(w) evaluated at the point w(n). The matrix H(n) is the m-by-m Hessian matrix of E(w) (second derivative), H = ²E(w)
Practical Techniques for Improving Backpropagation n n n H = ²E(w) requires the cost function E(w) to be twice continuously differentiable with respect to the elements of w. Differentiating Ew(n) g. T(n) w(n) +½ w. T(n) H(n) with respect to w, the change E(n) is minimized when g(n) + H(n) w(n) = 0 -> w(n) = H-1(n)g(n) w(n+1) = w(n) + w(n) w(n+1) = w(n)+H-1(n)g(n) where H-1(n) is the inverse of the Hessian of E(w).
Practical Techniques for Improving Backpropagation Newton’s method converges quickly asymtotically and does not exhibit the zigzagging behavior. n Newton’s method requires that the Hessian H(n) has to be a positive definite matrix for all n! n
Practical Techniques for Improving Backpropagation 3) Gauss-Newton Method n It is applicable to a cost function that is expressed as the sum of error squares. n E(w) = ½ i=1 n e²(i), note that all the error terms are calculated on the basis of a weight vector w that is fixed over the entire observation interval 1 i n. n The error signal e(i) is a function of the adjustable weight vector w. Given an operating point w(n), we linearize the dependence of e(i) on w by writing e’(i, w) = e(i) + [ e(i)/ w]Tw=w(n) (w-w(n)), i=1, 2, . . . , n n
Practical Techniques for Improving Backpropagation e’(n, w) = e(n) + J(n)(w-w(n)) where e(n) is the error vector e(n) = [e(1), e(2), . . . , e(n)]T and J(n) is the n-by-m Jacobian matrix of e(n) (The Jacobian J(n) is the transpose of the m-by-n gradient matrix e(n), where e(n) =[ e(1), e(2), . . . , e(n)]. w(n+1) = arg min w {½ e’(n, w) ²} = ½ e(n) ² +e. T(n)J(n)(w-w(n)) + ½(ww(n))TJT(n)J(n)(w-w(n)) Differentiating the expression with respect w and setting the result equal to zero
Practical Techniques for Improving Backpropagation JT(n)e(n) + JT(n)e(n) (w-w(n)) = 0 w(n+1) = w(n) – (JT(n)J(n))-1 JT(n)e(n) The Gauss-Newton requires only the Jacobian matrix of the error function e(n). For the Gauss-Newton iteration to be computable, the matrix product JT(n)J(n) must be nonsigular. JT(n)J(n) is always nonnegative definite but to ensure that it is nonsingular, the Jacobian J(n) must have row rank n. -> add the diagonal matrix I to the matrix JT(n)J(n), the parameter is a small positive constant.
Practical Techniques for Improving Backpropagation JT(n)J(n)+ I ; positve definite for all n. n -> The Gauss-Newton method is implemented in the following form: w(n+1) = w(n) – (JT(n)J(n) + I )-1 JT(n)e(n) n This is the solution to the modified cost function: n E(w) = ½{ w-w(0) ²+ i=1 n e²(i)} n where w(0) is the initial value of w. n
Regularization, Complexity Adjustment and Pruning n n n If we have too many weight and train too long > overfitting Wald Statistic: we can estimate the importance of a parameter in a model The Optimal Brain Damage, The Optimal Brain Surgeon: eliminate the parameter having the least importance
Summary n n Multilayer nonlinear neural networks trained by gradient descent methods such as backpropagation perform a maximum-likelihood estimation of the weight values in the model defined by the network topology. f(net) at the hidden units allows the networks to form an arbitary decision boundary, so long as there are sufficiently many hidden units.
- Multilayer neural network
- Bcmsn
- Multilayer perceptron nonlinear
- Multilayer mirror
- Multilayer architecture
- Multilayer perceptron
- Strategic multilayer assessment
- Multilayer security architektur
- Multilayer fabric
- Multilayer inspection firewall
- Multilayer pcb
- Multilayer ceramic capacitor
- Packet tracer firewall icon
- Multilayer sdn
- Multilayer perceptron calculation example
- 01:640:244 lecture notes - lecture 15: plat, idah, farad
- Graph neural network lecture
- Convolutional neural network alternatives
- Neural networks and learning machines
- Convolutional neural networks
- Pixelrnn
- 11-747 neural networks for nlp
- Audio super resolution using neural networks
- Neuraltools neural networks
- Tlu in neural network
- Mippers
- Visualizing and understanding convolutional networks
- Few shot learning with graph neural networks
- Matlab u-net
- Neural networks simon haykin
- Convolutional neural networks for visual recognition
- Gated recurrent unit in deep learning
- The wake-sleep algorithm for unsupervised neural networks
- Neural networks and fuzzy logic
- Convolution neural network ppt
- Vc dimension neural network
- Deep forest: towards an alternative to deep neural networks
- Sparse convolutional neural networks