Going Backwards In The Procedure and Recapitulation of

Going Backwards In The Procedure and Recapitulation of System Identification By Ali Pekcan 65570 B T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems

Outlines o Going Backwards in the Procedure n n n o Training the Network Again Finding the Optimal Network Architecture Optimal Brain Surgeon Optimal Brain Damage Section Summary Recapitulation of System Identification 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 2

Going Backwards in the Procedure o o Necessary to go back in the procedure if a model is not accepted immediately. “Stepping back” n n 1/9/2022 Dissatisfaction with the network model Possible better model T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 3

Training the Network Again o o Retrain the network with a different initialization of the weights. Retrain the network according to another criterion. The generalization ability of a model structure depends on the weight decay and minimum of the optimal generalization. Determination of the optimal weight decay and finding the global minimum: n n 1/9/2022 If there is a test set: Perhaps the best way is trial-and-error test. If a test set is not available: useful to evaluate average generalization error for the trained networks in terms of FPE and LULOO estimates. T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 4

Finding the Optimal Network Architecture (1) o Two fundamental choices: n n o o The contents of the regression vector What weights to include in the network architecture The goodness of the structure can be ranked by the average generalization error. Common procedure: Select a particular structure of the regression vector (NNARX, NNARMAX, NNOE, …) n Try to determine the best possible network architecture for this choise of regressors. More training data less important architecture determination. n 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 5

Finding the Optimal Network Architecture (2) o o For large amount of training data fully connected networks can be used and architecture selection is reduced to a matter of choosing the number of hidden units. Increasing the number of hidden units gradually while evaluating the test error can be a good way to choose. If the training set is very limited, the most essential weights should be chosen. (pruning algorithms) Principle of Pruning Algorithms: Eliminating the weights one at a time, of an architecture that is large enough to describe the system, until the optimal architecture is reached. 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 6

Comparing a fully connected NNARX(2, 2, 1) model structure with 91 weights to a model structure obtained by pruning the network until it contained 19 weights 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 7

Finding the Optimal Network Architecture (3) o Eliminating the weights (Ranking): n n Minimization of the unregularized criterion gives an asymptotically Gaussian distributed estimate of the weights where Obvious candidates for elimination are naturally those weights for which zero is within a distance of 2 -3 standard deviations from the estimates, . (Hassibi and Stork 1993) The way of ranking these candidates: comparing the weights to its standard deviation. If the ratio is small variance of weight is large maximum decrease in Average Generalization 1/9/2022 T-61. 6050 Neural Networks for Modelling and 8 Error Control of Dynamic Systems n

Optimal Brain Surgeon (OBS) -(1) o o First Derived for unregularized mean square error criteria, and was not directly motivated by the generalization error considerations. (Hassibi and Stork (1993)) The extension to the regularized case and connection to the average generalization error. (Pedersen and Hansen (1995)) For a single output network model trained to its minimum, the FPE estimate of the average generalization error for the unregularized criterion is: If a regularization term is included Little accuracy difference is expected between 3 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 9

Optimal Brain Surgeon (OBS) -(2) o o o OBS is to eliminate the weight that gives a maximum decrease in the FPE estimate. Each weight affects the FPE: Let Mj specify the model structure with the jth weight Second equation is Saliency for weight j : Change in training error n Third equation is generalization-based saliency for weight j : Change in the average generalization error. OBS is to eliminate the weight with the smallest generalization-based saliency n 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 10

Optimal Brain Surgeon (OBS) -(3) o In the regularized case it is necessary to find the minimizer of; where is the effective number of weights in Mj o Computing the Salience: 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 11

Optimal Brain Surgeon (OBS) -(4) o since 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 12

Optimal Brain Surgeon (OBS) -(5) o o Without weight decay the first two terms will be zero Like the ranking ratio before 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 13

Optimal Brain Surgeon (OBS) -(6) o Problem: In over parameterized networks Hessian of the unregularized criterion, R, generally is ill-conditioned. So the inverse Hessian should be approximated with the recursion used in the recursive Gauss-Newton algorithm. Weight decay improves the robustness of the OBS procedure. The effective number of weights can be approximated by : o And the inverse Hessian as the Schur complement of Q o o o 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 14

Optimal Brain Surgeon (OBS) -(7) o For two layer network: n n n In the beginning only input-to-hidden layer weights are eliminated When a hidden unit has only one weight leading to it, the saliency for entire unit is determined If this unit saliency is smaller than any other saliency, the unit is eliminated. Its incorrect to calculate saliency as the change in FPE obtained by fixing both weights connected to the unit to zero. J=(j 1, j 2) contain the locattions of two weights. 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 15

Complete procedure for pruning by (OBS) 1. 2. 3. 4. 5. 6. 7. 1/9/2022 Select a sufficiently large model structure and train the network with small weight decay. Compute the Gauss-Newton Hessian and invert it. Evaluate FPE estimate and evaluate the test error if a test set is available. If both quantities pass respective the minimum go to Step 7 Compute the saliency for each weight, effective number of parameters in the reduced network and combine them into generalization-based saliencies. Search for the smallest generalization-based saliency and eliminate the concerned weight(s) from the parameter vector. Determine the remaining weights. Retrain, store the weights and go to Step 2 (Levenberg-Marquardt 10 -50 iterations) Re-establish the architecture that was optimal as assessed by the test error and FPE estimate, and retrain the network without weight decay T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 16

Optimal Brain Damage (OBD) o o o o It’s based on the assumption that all off-diagonal elements in the Hessian matrix can be neglected. Unlike OBS, OBD only provides guidelines for pruning and not for how to reestimate the remaining weights. When OBD removes a weight, it leaves the remaining weights untouched. Not modifies like OBS. The saliency of weight j is given by : More than one weight at a time is allowed. Computation is less than OBS and even performs similarly. Most demanding part is to find the initial model structure which is large enough to describe the system. Generalization-based saliency of weight j : 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 17

Section Summary o o o Retraining the network The found minimum can be quite far in value from the global minimum so the network should be retrained with different initializations of the weights. Not necessary to validate first. Selecting the optimal model structure OBS-OBD both prune the network weights of one at a time until optimal architecture is reached. Stopping criterion one or more average generalization error (FPE, LULOO) OBD Only prunes weights OBS Prunes weights and recommends how to change the value of the remaining weights in the network. Continuing the experiment If its not possible to describe the system satisfactory with the available training data, it may be necessary to redo the experiment. 1/9/2022 T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 18

Recapitulation of System Identification o o Experiment collect as much data as possible, with visual inspection Model structure selection n o Small amount of data fully connected NNARX models are recommended Medium amount of data (bias and variance error is found) regularization by weight decay and pruning methods are discussed. Large amount of data (without regularization term fully connected nets) Estimate a model n n n 1/9/2022 Select a criterion specify how the weights should be determined from data. Choose an iterative search method for minimizing the criterion. (training algorithm) Ex: Levenberg Marquardt, Quasi Newton methods T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 19

Recapitulation of System Identification o Validation n 1/9/2022 Correlation tests to investigate if the residuals are white and independent of past information One-step and k-step ahead predictions, estimation of prediction intervals Estimation of the average generalization error T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 20

Recapitulation of System Identification o Going Backwards in the procedure o Retrain the network Network usually have more than one local minimum. 5 -7 times with different initialization of the weights o Determine optimal weight decay for regularization trial-anderror or a combination of FPE, LULOO, and test error to find the optimal weight decay. o Determine optimal model structure by pruning n n 1/9/2022 Determine a weight decay that is smaller than the optimal. Prune one weight at a time and let it be followed by retraining T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems 21

-End- T-61. 6050 Neural Networks for Modelling and Control of Dynamic Systems