RECURRENT NEURAL NETWORKS OUTLINE Why use recurrent networks

RECURRENT NEURAL NETWORKS

OUTLINE • Why use recurrent networks • Typical architectures • A taste of backprop 2

Simple Recurrent Network 3

Robot sensory prediction with SRN 4

Short Term Memory (STM) & Simple T-maze problems What types of neural nets have already been used for similar tasks and why? What are the inputs and the outputs in an artificial neural net used for such a task? Delayed response tasks are a standard way of investigating short-term memory (STM). The agent is typically assumed to ’remember’ in some way the necessary information about the stimulus (for example the side on which a stimulus appeared) during the delay period. Studies upto now are far away from suggesting a specific model of how animals solve delayed response tasks (Ziemke). The solution seems to lie in an interdisciplinary approach. Figure The two situations in the simple T-maze environment. Adapted from Ulbricht (1996). 5

Sensors of the Khepara Robot as Artificial Neural Network Inputs I 3 I 4 I 2 L 1 I 5 G I 6 L 2 Sensors of the Khepara robot used in this study: I 1, I 2. . I 6: infrared sensors; G: ground (zone) sensor. L 1, L 2: Light sensors. 6

Repeated T-maze 7

Multiple T-maze 8

Ziemke’s Food/Poison Maze (1999) 9

Recurrent Connections and Sequences • A sequence is a succession of patterns that relate to the same object. • For example, letters that make up a word or words that make up a sentence. • Sequences can vary in length. This is a challenge. • How many inputs should there be for varying length inputs? • Several feed-forward alternatives: shift registers, tapped delay lines, etc 10

The simple recurrent network • Jordan network has connections that feed back from the output to the input layer and also some input layer units feed back to themselves. • Useful for tasks that are dependent on a sequence of a successive states. • The network can be trained by backpropogation. • The network has a form of short-term memory. • Simple recurrent network (SRN) has a similar form of short-term memory. 11

Figure 5. 6 12

Elman Network (SRN). The number of context units is the same as the number of hidden units 13

Modified Elman Network (Adapted from Rylatt and Czarnecki (2000)) In addition predict sensory inputs. 14

Short-term memory in SRN • The context units remember the previous internal state. • Thus, the hidden units have the task of mapping both an external input and also the previous internal state to some desired output. 15

• An SRN can predict the next item in a sequence from the current and preceding input. • In an SRN the hidden units are fed back into the input layer. • That is: The hidden units are copied into input units. Input units supply weighted sums into hidden units in the next time step. ********** • The hidden units represent an internal reduced representation of the data in the sequence that precedes the current input. 16

• The reduced representation provides context which is essential for certain tasks. • As an example SRN can learn to solve XOR problem. • Input – 101000011110101…. . – Bit 3 is XOR of bit 1 and 2, bit 6 is XOR of 4 and 4, and so on • An SRN with 1 input, two context, two hidden and one output unit was trained on a sequence of 3000 bits. input: 1 0 0 0 0 1 1 0 1. . . output: 0 1 0 0 1 1 0 1 ? . . . 17

18

ESCN (Extended Sequential Cascaded Networks) & T-mazes Typically used networks for STM tasks are first-order or higher-order feedback recurrent networks. In first-order recurrent networks, certain neuron activation values are fed back into the network, typically in the next time step such as to remember agent’s goal, which part of the environment agent is currently in etc. In the case of higher-order feedback, its typically connection weights (and/or bias weights) that are modulated dynamically. Examples are sequential cascaded networks (SCN) or Extended(SCN). 19

ESCN More specifically, given an input vector ij(t), j = 1…n, state unit vector s(t) and output vector o(t) are usually calculated as follows by the function network: Where f is the logistic activation function, and W 0(t) and WS(t), together referred to as function network weights, are two-dimensional connection weight matrices dynamically computed in every time step by the context network. W 0(t) = {wa(t), wb(t), ……. wn(t) } // ONE FOR EACH OUTPUT UNIT Ws(t) = {wsa(t), wsb(t), ……. wsn(t) } // ONE FOR EACH STATE UNIT 20

ESCN There’s an additional decision unit in ESCN. Given the state vector sk(t), k = 1…m, the decision unit activation d(t) and the function network weight matrices W 0(t) and WS(t), are dynamically computed in every time step t as follows by the ESCN’s context network: Where f is the logistic activation function, Vd is a one-dimensional connection weight matrix, mapping current state to decision unit activation, and V 0 and VS are two-dimensional connection weight matrices mapping the current internal state s(t) to the next time step’s function network weights, if the decision unit is active, i. e. has an activation value of at least 0. 5. 21

ESCN Example Behaviour (Ziemke 1999) 22

BP: Unfolding Network Example • A 3 -2 -2 fully connected network has recurrent connections where a hidden unit connects to itself and to every other output unit. Sketch the network and its equivalent feedforward network with no recurrent connections, for a single time step. The single time step in this example means that an input pattern will have passed between two hidden units and two output units. 23

Figure 5. 4 24

Figure 5. 5 25

A recurrent backpropogation network • A backpropogation network need not be strictly feedforward and can have recurrent connections. • A unit can feed to itself, to units in the same or lower levels. • A recurrent connection feeds back activation that will affect the output from the network during subsequent iterations. • For every recurrent network there is a feedforward network with identical behavior. 26

Figure A recurrent backpropogation network (Connections from output to input layer) 27

An expanded version of the network in the previous figure. 28

Backprop Through Time • During training a pattern is presented to the network and a feedforward pass is made. • Each network copy corresponds to a time step. • The weight changes calculated for each network copy are summed before individual weights are adapted. • The set of weights for each copy(time step) always remain the same. 29

Sequence Completion Example • Rumelhart et. al. (1986) performed experiments with recurrent connections. • Train a network to perform sequence completion. • A sequence consisted of 6 characters: two letters from the set {A, B, C, D, E} and 4 being numbers from the integer set {1, 2, 3}. • For example, A can denote “ 12” and C can denote “ 13”. • The complete sequence prefixed by AC would be “AC 1213”. • The complete sequence prefixed by AA would be “AA 1212”. 30

Rumelhart network • The network has: – 5 input units, one for each letter. – 3 output units, one for each number. – 30 hidden units. Each hidden unit is connected to itself and every other hidden unit. 31

Training Steps T 1: Input unit corresponding to the first letter is 1 and other inputs are off. T 2: Input unit corresponding to the second letter is 1 and other inputs are off. T 3: The target output is the third character in the sequence. T 4: The target output is the fourth character in the sequence. T 5: The target output is the fifth character in the sequence. T 6: The target output is the sixth character in the sequence. Output generation at different time steps. 32

Rumelhart Network Results • All hidden and output unit activations started at 0. 2 • Network is trained on 20 sequences. • Given first two characters, the rest of the characters were completed in five test cases. • The completion could be achieved even if there were delays between the presentation of the first two letters. 33

Could the study of recurrent networks lead to studying other capabilites of memory? Yes, counting. 34

An example: Counting to three. 35

Evolving Virtual Creatures • Karl Sims (1994) • ANN Controllers • Both body and ”mind” evolved • http: //web. genarts. co m/karl/ 36