Neural Networks Background Neural Networks can be Biological
- Slides: 45
Neural Networks
Background - Neural Networks can be : - Biological models - Artificial models - Desire to produce artificial systems capable of sophisticated computations similar to the human brain.
Biological analogy and some main ideas • The brain is composed of a mass of interconnected neurons – each neuron is connected to many other neurons • Neurons transmit signals to each other • Whether a signal is transmitted is an all-or-nothing event (the electrical potential in the cell body of the neuron is thresholded) • Whether a signal is sent, depends on the strength of the bond (synapse) between two neurons
How Does the Brain Work ? (1) NEURON - The cell that performs information processing in the brain. - Fundamental functional unit of all nervous system tissue.
How Does the Brain Work ? (2) Each consists of : SOMA, DENDRITES, AXON, and SYNAPSE.
Brain vs. Digital Computers (1) - Computers require hundreds of cycles to simulate a firing of a neuron. - The brain can fire all the neurons in a single step. Parallelism - Serial computers require billions of cycles to perform some tasks but the brain takes less than a second. e. g. Face Recognition
Definition of Neural Network A Neural Network is a system composed of many simple processing elements operating in parallel which can acquire, store, and utilize experiential knowledge.
Artificial Neural Network? Neurons vs. Units (1) • Each element of NN is a node called unit. • Units are connected by links. • Each link has a numeric weight.
Neurons vs. units (2) Real neuron is far away from our simplified model - unit Chemistry, biochemistry, quantumness.
Computing Elements A typical unit:
Planning in building a Neural Network Decisions must be taken on the following: - The number of units to use. - The type of units required. - Connection between the units.
How NN learns a task. Issues to be discussed - Initializing the weights. - Use of a learning algorithm. - Set of training examples. - Encode the examples as inputs. - Convert output into meaningful results.
Neural Network Example A very simple, two-layer, feed-forward network with two inputs, two hidden nodes, and one output node.
Simple Computations in this network - There are 2 types of components: Linear and Nonlinear. - Linear: Input function - calculate weighted sum of all inputs. - Non-linear: Activation function - transform sum into activation level.
Calculations Input function: Activation function g:
A Computing Unit. Now in more detail but for a particular model only A unit
Activation Functions - Use different functions to obtain different models. - 3 most common choices : 1) Step function 2) Sign function 3) Sigmoid function - An output of 1 represents firing of a neuron down the axon.
Step Function Perceptrons
3 Activation Functions
Standard structure of an artificial neural network • Input units – represents the input as a fixed-length vector of numbers (user defined) • Hidden units – calculate thresholded weighted sums of the inputs – represent intermediate calculations that the network learns • Output units – represent the output as a fixed length vector of numbers
Representations • Logic rules – If color = red ^ shape = square then + • Decision trees – tree • Nearest neighbor – training examples • Probabilities – table of probabilities • Neural networks – inputs in [0, 1] Can be used for all of them Many variants exist
Notation
Notation (cont. )
Operation of individual units • Outputi = f(Wi, j * Inputj + Wi, k * Inputk + Wi, l * Inputl) – where f(x) is a threshold (activation) function – f(x) = 1 / (1 + e-Output) • “sigmoid” – f(x) = step function
Artificial Neural Networks
Perceptron Learning Theorem • Recap: A perceptron (threshold unit) can learn anything that it can represent (i. e. anything separable with a hyperplane) 26
The Exclusive OR problem A Perceptron cannot represent Exclusive OR since it is not linearly separable. 27
28
Properties of architecture • No connections within a layer • No direct connections between input and output layers • Fully connected between layers • Often more than 3 layers • Number of output units need not equal number of input units • Number of hidden units per layer can be more or less than input or output units Each unit is a perceptron Often include bias as an extra weight 29
Conceptually: Forward Activity Backward Error 30
Backpropagation learning algorithm ‘BP’ Solution to credit assignment problem in MLP. Rumelhart, Hinton and Williams (1986) (though actually invented earlier in a Ph. D thesis relating to economics) BP has two phases: Forward pass phase: computes ‘functional signal’, feed forward propagation of input pattern signals through network Backward pass phase: computes ‘error signal’, propagates the error backwards through network starting at output units (where the error is the difference between actual and desired output values) 31
Forward Propagation of Activity • Step 1: Initialize weights at random, choose a learning rate η • Until network is trained: • For each training example i. e. input pattern and target output(s): • Step 2: Do forward pass through net (with fixed weights) to produce output(s) – i. e. , in Forward Direction, layer by layer: • • • Inputs applied Multiplied by weights Summed ‘Squashed’ by sigmoid activation function Output passed to each neuron in next layer – Repeat above until network output(s) produced 32
Step 3. Back-propagation of error 33
‘Back-prop’ algorithm summary (with Maths!) 34
‘Back-prop’ algorithm summary (with NO Maths!) 35
MLP/BP: A worked example 36
Worked example: Forward Pass 37
Worked example: Forward Pass 38
Worked example: Backward Pass 39
Worked example: Update Weights Using Generalized Delta Rule (BP) 40
Similarly for the all weights wij: 41
Verification that it works 42
Training • This was a single iteration of back-prop • Training requires many iterations with many training examples or epochs (one epoch is entire presentation of complete training set) • It can be slow ! • Note that computation in MLP is local (with respect to each neuron) • Parallel computation implementation is also possible 43
Training and testing data • How many examples ? – The more the merrier ! • Disjoint training and testing data sets – learn from training data but evaluate performance (generalization ability) on unseen test data • Aim: minimize error on test data 44
More resources • Binary Logic Unit in an example – http: //www. cs. usyd. edu. au/~irena/ai 01/nn/5. html • Multi. Layer Perceptron Learning Algorithm – http: //www. cs. usyd. edu. au/~irena/ai 01/nn/8. html 45
