Simple recurrent networks Today Whats so great about

Simple recurrent networks.

Today • What’s so great about backpropagation learning? • Temporal structure and simple recurrent networks—basic concepts • Examples applied in language and cognitive neuroscience

What’s so great about backpropagation learning?

b 3 Input 1 Input 2 Output 0 0 0 1 0 1 1 0 w 1 w 2 1 2 1 0. 8 1 0 0. 6 0. 4 0. 2 0 0 -0. 2 0 1 -0. 4 -0. 6 0. 2 0. 4 0. 6 0. 8 1

5 w 6 3 4 w 2 w 1 1 w 3 w 4 2

01 11 Unit 3 Input space 01 11 00 10 01 11 Unit 4 00 00 10 10 Hidden space Input space 01 00 11 11 10 00

So backpropagation in a deep network (1 or more hidden layers) allows the network to learn to re-represent the structure provided in the input in ways that suit the task…

Rumelhart and Todd (1993) Item Attributes living pretty green big small bright dull… pine oak rose daisy robin canary sunfish salmon grow fly sing swim… Context ISA is… can… has… leaves roots petals wings feathers scales gills…

Epoch 150 Pine Oak Rose Daisy Robin Canary Sunfish Salmon

Rumelhart and Todd (1993) Attributes living pretty green big small bright dull… pine oak rose daisy robin canary sunfish salmon move fly sing swim… Item Context ISA is… can… has… leaves roots petals wings feathers scales gills…

Patterns that emerge in hidden layer will be influenced by overlap in both the input units and the output units… Generalization of learning from one item to another depends on how similar the items are in the internal learned representation… So backprop learning provides one way of thinking about how appropriately-structured internal representations might be acquired.

Surprise test! Spell MISSISSIPPI

Human behavior occurs over time • • • Speaking and comprehending Making a cup of coffee Walking Seeing Essentially everything How does any information processing system cope with time?

Spoken output Visual input 1 2 3 4 5 6 A B C D. . . Z A B C D. . . Z 1 2 3 4 5 A B C D. . . Z 6 … A B C D. . . Z 11 A B C D. . . Z … 11

Left-justified 1 L G S 2 O L P 3 4 5 G A D L I T Vowel-centred -3 -2 -1 S S W S P L 0 U A I 1 N M T

“Jordan” network COPY (t-1)

Simple Recurrent Network (SRN) Current output (t) Hidden units Copy (t-1) Hidden rep at t-1 Current input (t)

XOR in time! 0 1 1 0 0 0 1 1 0 COPY (t-1) 1

Network can combine current input plus “memory” of previous input to predict next input…

What about “longer” memory, more complex patterns? ba dii guuu

Error Time step

Simple Recurrent Network (SRN) Current output (t) Hidden units Copy (t-1) Hidden rep at t-1 Current input (t)

SRN can “hold on” to prior information over many time steps!

But that’s not all…

Error Time step

Consonantal High

SRN “knows” about consonants, sort of!

Learning word boundaries • 15 words • 200 sentences (4 -9 words in length) • Broken into letter parts • Each letter is a random bit vector • Predict next letter

Maybe predictive error can serve as a cue for parsing the speech stream (and other sequential behaviors!)

What about word categories?

• Localist representation of 31 words • Series of 2 -3 word sentences strung together: – woman smash plate cat move man break car boy move girl…. • Predict next word…

Simple Recurrent Network (SRN) Current output (t) Hidden units Copy (t-1) Hidden rep at t-1 Current input (t)

ZOG break car Zog eat Zog like girl ZOG chase mouse…

Words in context…

SRN acquires representational structure based on implicit temporal prediction… Items represented as similar when preceded or followed by similar distributions of items

Example in cognitive neuroscience • Hypothesis from literature: Events are parsed when transition probabilities are low… • Alternative hypothesis….

Example in cognitive neuroscience • Hypothesis from literature: Events are parsed when transition probabilities are low… • Alternative hypothesis: events are parsed at transitions between communities

So SRNs… • Can implicitly learn structure of temporally extended sequences… • Can provide graded cues to “segmenting” such sequences… • Can learn similarities amongst items that participate in such sequences, based on the similarity of their temporal contexts.

Example applications • Working memory: n-back, serial order, AX task. • Language parsing • Sequential word processing • Sentence processing • Routine sequential action