Outline Neural networks Multilayer neural networks continued Backpropagation

Outline • Neural networks – Multi-layer neural networks - continued • Back-propagation – Applications 11/25/2020 Visual Perception Modeling 1

Simple Perceptrons • Simple perceptrons – One-layer feed-forward network • There is an input layer and an output layer and no hidden layers – The computation can be described by • Thresholds are omitted because they can always be treated as connections to an input terminal that is – 1 permanently 11/25/2020 Visual Perception Modeling 2

A Simple Learning Algorithm • There is a learning algorithm for a simple perceptron network – Given a training pattern k , the desired output is i – The learning algorithm, or the procedure to change its weights, is 11/25/2020 Visual Perception Modeling 3

Hebbian Learning Rules • Hebbian learning – The strength of a synaptic connection should be adjusted if its level of activity changes – An active synapse which repeatedly triggers the activation of its postsynaptic neuron will grow in strength • Anti-Hebbian learning rule 11/25/2020 Visual Perception Modeling 4

Simple Perceptrons – cont. • Linear separability – For simple perceptrons, the condition for correct operation is that a plane should divide the inputs that have positive and negative targets – This means the decision boundary will be a plane where • The plane is w • = 0 11/25/2020 Visual Perception Modeling 5

Simple Perceptrons – cont. • Linear units • Gradient descent learning 11/25/2020 Visual Perception Modeling 6

Multi-layer Perceptrons • The limitations of perceptrons do not apply to feed-forward networks with hidden layers between the input and output layer with nonlinear activation function • The problem is to train the network efficiently 11/25/2020 Visual Perception Modeling 7

Multi-layer Perceptrons – cont. 11/25/2020 Visual Perception Modeling 8

Multi-layer Perceptrons – cont. • Back-propagation – Extension of the gradient descent learning rule – The hidden-to-output layer connections 11/25/2020 Visual Perception Modeling 9

Multi-layer Perceptrons – cont. • Back propagation - continued – Input-to-hidden connections where 11/25/2020 Visual Perception Modeling 10

Activation Function • Activation function – For back-propagation, the activation function must be differentiable – Also we want it to saturate at both extremes – Sigmoid function 11/25/2020 Visual Perception Modeling 11

Activation Function – cont. 11/25/2020 Visual Perception Modeling 12

Back Propagation Algorithm 1. 2. 3. 4. 5. Initialize the weights to small random values Choose a pattern ku and apply it to the input layer Propagate the signal forward through the network Compute the deltas (errors) for the output layer Compute the deltas (errors) for the preceding layers by propagating the errors backwards 6. Update all the connections according to the algorithm 7. Go back to step 2 and repeat for the next pattern 11/25/2020 Visual Perception Modeling 13

Using Neural Networks • Design phase – The neural network architecture • Training phase – Use available examples to train the neural network • That is, to use the back-propagation algorithm to learn the connection weights • Test phase – For a new sample, feed the feature through the neural network and you go the result 11/25/2020 Visual Perception Modeling 14

Other Neural Network Models • Hopfield model – associate memory model – You can store many patterns in one neural network – Then the network will recall the “correct” pattern based on the input • There are many other kinds of neural networks – They are generally designed for a more specific problem 11/25/2020 Visual Perception Modeling 15

Applications • Application examples – NETtalk – Navigation of a car – Image compression – Recognizing hand-written ZIP codes – Speech recognition – Face recognition 11/25/2020 Visual Perception Modeling 16

Back Propagation Program • Programs – Backprop. c – main program – Propagation. c – contains procedures for BP – Para-util. h and type-def. h – contain data structure definitions – Located at ~liux/public_html/courses/research/programs/neural-networks • Parameter files – Control parameter file – network-3 -3 -1. par – Training data file – network-3 -3 -1 -training. par 11/25/2020 Visual Perception Modeling 17