Nanjing University of Science Technology Pattern Recognition Statistical

Nanjing University of Science & Technology Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 24 Nov 2, 2005 1

Lecture 24 Topics 1. Review and Motivation for Link Structure 2. Present the Functional Link Artificial Neural Network. 3. Simple Example- design using ANN and FLANN 4. Performance for Neural Network Designs 5. Radial Basis Function Neural Networks 6. Problems, Advantages, Disadvantages, and promise of Artificial Neural Network Design 2

Generalized Linear Discriminant Functions w 1 g 1(x) x … g 2(x) … gj(x) g. M(x) Review 1 w 2 x x wj x + g(x) w. M x 3

Patterns are linearly separated in the 3 -dim space Separating plane Review 2 4

Example: Decision rule using one nonlinear discriminant function g(x) Given the following g(x) and decision rule Illustrate the decision regions R 1 and R 2 where we respectively classify as C 1 and C 2 for the decision rule above Review 3 5

Solution: R 1 decide C 1 R 2 decide C 2 every where else Review 4 6

Find a generalized linear discriminant function that separates the classes Solution: d(x) = w 1 f 1(x)+ w 2 f 2(x)+ w 3 f 3(x) + w 4 f 4(x) +w 5 f 5(x) + w 6 f 6(x) T = w f (x) in the f space (linear) Review 5 7

where in the original pattern space: (nonlinear) Review 6 8

Decision Boundary in original pattern space x 2 from C 1 2 from C 2 1 1 -1 -2 Review 7 2 3 4 x 1 Boundary d(x) = 0 9

Potential Function Approach – Motivated by electromagnetic theory + from C 1 - from C 2 Sample space Review 8 10

Plot of Samples from the two classes Review 9 11

Given Samples x from two classes C 1 and C 2 S 1 S 2 C 1 C 2 Define Total Potential Function K(x) = ∑ K(x, xk) - ∑ K(x, xk) xk C S 1 xk C S 2 Decision Boundary Review 10 Potential Function K(x) = 0 12

Algorithm converged in 1. 75 passes through the data to give final discriminant function as Review 11 13

Functional Link Neural Network 14

Quadratic Functional Link 15

Fourier Series Functional Link 16

Principal Component Functional Link fk(x), k=1 to N are chosen as the eigen vectors of the sample covariance matrix 17

Example: Comparison of Neural Net and functional link neural net Given two pattern classes C 1 and C 2 with the following four patterns and their desired outputs 18

(a)Design an Artificial Neural Network to classify the two patterns given (b) Design a Functional Link Artificial Neural Network to classify the patterns given. (c) Compare the Neural Net and Functional Link Neural Net designs 19

(a) Solution: Artificial Neural Net Design Select the following structure 20

After training using the training set and the backpropagation algorithm the design becomes Values determined by neural net 21

(b) Solution: Functional Link Artificial Neural Net Design 22

A neural net was trained using the functional link output patterns as new pattern samples The resulting weights and structure are 23

(c) Comparison Artificial Neural Net (ANN) and Functional Link Artificial Neural Net (FLANN} Designs FLANN has simpler structure than the ANN with only one neural element and Link. Fewer iterations and computations in the training algorithm for FLANN design may be more sensitive to 24 errors in patterns.

Determining Performance of Neural Net Design on Training Set 25

Determine Performance for Design using Training Set Classify each member of the training set using the neural network design. Test Design on Testing Set Classify each member of the testing set using the neural network design. 26

Could use (a) Performance Measure ETOT (b) The Confusion Matrix (c) Probability of Error (d) Bayes Risk 27

(a) Local and global errors- Used in Neural Net Design procedure Local Measure Global Measure 28

(b) Confusion Matrix- Example Correct Classification Incorrect Classification 29

(c) Probability of Error- Example Estimates of Probabilities of being Correct Estimate of Total Probability of Error 30

(d) Bayes Risk Estimate 31

Radial Basis Function (RBF) Artificial Neural Network Functional Link 32

Functional Form of RBF ANN where Examples of Nonlinearities 33

Design Using RBF ANN Let F(x 1, x 2, … , xn) represent the function we wish to approximate. For pattern classification F(x) represents the class assignment or desired output (target value) for each pattern vector x a member of the training set Define the performance measure E by E We wish to Minimize E by selecting a , b 1, b 2, . . . , b. M and z 1, z 2, . . . z. M M, 34

Finding the Best Approximation using RBF ANN Usually broken into two parts (1 st ) Find the number M of prototypes and the prototypes { zj : j=1, 2, . . . , M} by using a clustering algorithm(Presented in Chapter 6) on the training samples (2 nd ) With these fixed M and { zj: j=1, 2, . . . , M} find the a , b 1, b 2, . . . , b. M that minimize E. Notes: You can use any minimization procedure you wish. Training does not use the Backpropagation Algorithm 35

Problems Using Neural Network Designs Failure to converge Selection of insufficient structure Max iterations too small Lockup occurs Limit cycles Good performance on training set – poor performance on testing set Training set not representative of variation Too strict of a tolerance - “grandmothering” 36

Advantages of Neural Network Designs Can obtain a design for very complicated problems. Parallel structure using identical elements allows hardware or software implementation Structure of Neural Network Design similar for all problems. 37

Other problems that can be solved using Neural Network Designs System Identification Functional Approximation Control Systems Any problem that can be placed in the format of a clearly defined desired output for different given input vectors. 38

Famous Quotation “Neural network designs are the second best way to solve all problems” 39

Famous Quotation “Neural network designs are the second best way to solve all problems” ? ? ? ? ? 40

Famous Quotation “Neural network designs are the second best way to solve all problems” ? ? ? ? ? The promise is that a Neural Network can be used to solve all problems; however, with the caveat there is always a better way to solve a specific problem. 41

So what is the best way to solve a given problem ? ? ? 42

So what is the best way to solve a given problem ? ? ? ? 43

So what is the best way to solve a given problem ? ? ? ? A design that uses and understands the structure of the data !!! 44

Summary Lecture 24 1. Reviewed and Motivated Link Structure 2. Presented the Functional Link Artificial Neural Network. 3. Presented Simple Example with designs using ANN and FLANN 4. Described Performance Measures for Neural Network Designs 5. Presented Radial Basis Function Neural Networks 45

6. Discussed Problems, Advantages, Disadvantages, and the Promise of Artificial Neural Network Design 46

End of Lecture 24 47