CSCI 5922 Neural Networks and Deep Learning Representation

  • Slides: 26
Download presentation
CSCI 5922 Neural Networks and Deep Learning: Representation Mike Mozer Department of Computer Science

CSCI 5922 Neural Networks and Deep Learning: Representation Mike Mozer Department of Computer Science and Institute of Cognitive Science University of Colorado at Boulder

Representation ü Definition? § a formalism for capturing certain key features of some other

Representation ü Definition? § a formalism for capturing certain key features of some other domain ü E. g. , a square ü A representation makes certain domain information explicit (and easy to compute) and other information implicit (and difficult to compute)

Representing An English Word ü ü Localist or “one hot” encoding ABLE 1 0

Representing An English Word ü ü Localist or “one hot” encoding ABLE 1 0 0 0 AXLE 0 1 0 0 0 BOLT 0 0 0 1 0 0 0 0 0 GEAR 0 0 0 1 0 0 0 WEAR 0 0 0 0 0 1 0 0 0 Neutral representation ABLE § each item orthogonal to each other § no expected generalization from one item to another § favored by end-to-end deep learning folks BOLT AXLE

Representing An English Word ü Distributed orthographic representation A 1 B 1 . .

Representing An English Word ü Distributed orthographic representation A 1 B 1 . . . B 2 . . . X 2 A 3 … L 3 … E 4 R 4 T 4 ABLE 1 0 0 0 0 1 0 0 AXLE 1 0 0 1 0 0 BOLT 0 1 0 0 0 1 GEAR 0 0 0 1 0 WEAR 0 0 0 1 0 ABLE AXLE ü Captures similarity of spelling patterns § natural generalization from word to similarly spelled words BOLT WEAR GEAR

Representing An English Word ü Distributed semantic representation ABLE 1 0 0 0 0

Representing An English Word ü Distributed semantic representation ABLE 1 0 0 0 0 1 0 0 0 AXLE 0 1 1 0 0 0 1 0 BOLT 0 0 1 0 GEAR 0 1 1 0 0 0 0 1 1 WEAR 0 0 0 0 1 0 1 ABLE ü Captures similarity of meaning AXLE WEAR GEAR BOLT

Using Statistics of Natural Language To Discover Word Representations Useful For Other Tasks Neural

Using Statistics of Natural Language To Discover Word Representations Useful For Other Tasks Neural Probabilistic Language Model (Bengio, Ducharme, Vincent, 2003) Word 2 Vec (Mikolov, Chen, Corrado, Dean, 2013) Uses unlabeled data to determine representations

Comparison of Representations Local Distributed § Neuron activated for only one concept § One

Comparison of Representations Local Distributed § Neuron activated for only one concept § One neuron activated per concept § Neuron activated for many concepts § Many neurons activated per concept A 1 B 1 . . . B 2 . . . X 2 A 3 … L 3 … E 4 R 4 T 4 ABLE 1 0 0 0 0 0 ABLE 1 0 0 0 0 1 0 0 AXLE 0 1 0 0 0 0 AXLE 1 0 0 1 0 0 BOLT 0 0 0 1 0 0 0 BOLT 0 1 0 0 0 1 GEAR 0 0 0 1 0 0 0 0 1 0 WEAR 0 0 0 0 0 1 WEAR 0 0 0 1 0

Comparing Representations ü Local § ABLE ü Distributed § A/1, B/2, L/3, E/4 ü

Comparing Representations ü Local § ABLE ü Distributed § A/1, B/2, L/3, E/4 ü Semi-distributed or sparse § AB, BL, LE

How Many Concepts Can Be Represented Simultaneously? ü Local ü § all of them

How Many Concepts Can Be Represented Simultaneously? ü Local ü § all of them ü § ABLE, GEAR, WEAN Distributed § one § binding problem ü § A/1, G/1, B/2, E/2, A/3, L/3, E/4, R/4 ü Semi-distributed § AB, AR, BL, EA, GE, LE § depends on similarity § AN, AR, EA, GE, WE

How Many Neurons Are Required? ü Local § many ü § one per word

How Many Neurons Are Required? ü Local § many ü § one per word (~2000 four letter words) Distributed § few ü ü § one per letter position (104) Semi-distributed § intermediate § one per letter pair (262 = 676)

Is Similarity Structure Captured? ü Local ABLE § no: neutral representation ü Distributed BOLT

Is Similarity Structure Captured? ü Local ABLE § no: neutral representation ü Distributed BOLT AXLE § yes: overlap, vector dot product, etc. ü ABLE AXLE Semi-distributed BOLT ABLE § intermediate AXLE BOLT WEAR GEAR

Does Representation Support Generalization? ü Local § no § easy to build arbitrary mappings

Does Representation Support Generalization? ü Local § no § easy to build arbitrary mappings ü AT AN 1 2 ? ? AM IF 1 2 IT IN ? ? M/2 F/2 Distributed § yes § similar patterns ⇒ similar consequences § generalization of learning A/1 I/1 T/2 N/2

How Difficult Is It To Interpret Representation? ü Local § easy ü FINK. 41

How Difficult Is It To Interpret Representation? ü Local § easy ü FINK. 41 LINK. 45 FIND. 64 ü Distributed § hard F/1. 48 I/2. 92 N/3. 62 K/4. 23 L/1. 54 D/4. 95

Representing New Concepts ü Local § need to add neurons ü LINE LEFT LIFT

Representing New Concepts ü Local § need to add neurons ü LINE LEFT LIFT ü Distributed § need to specify a unique pattern of activation across the existing neurons L/1 I/2 N/3 E/4 E/2 F/3 T/4

Tensor Product Representation (Smolensky, 1990) letter ü position 1 2 3 4 5 6

Tensor Product Representation (Smolensky, 1990) letter ü position 1 2 3 4 5 6 7 … 1 0 0 0 … A 0 0 0 0 … B 0 0 0 0 … C 1 1 0 0 0 … D 0 0 0 0 … E 0 0 0 0 … F 0 0 0 0 … … … X 0 0 0 0 … Y 0 0 0 0 … Z 0 0 0 0 …

Representing Multiple Role-Filler Pairs position letter ü 1 2 3 4 5 6 7

Representing Multiple Role-Filler Pairs position letter ü 1 2 3 4 5 6 7 … A 0 1 0 0 0 … B 0 0 1 0 0 … C 1 0 0 0 … D 0 0 0 0 … E 0 0 0 0 … F 0 0 0 0 … … … … … X 0 0 0 0 … Y 0 0 0 0 … Z 0 0 0 0 …

Distributed Roles ü 1 2 3 § 1 . 5 1. 5 0 0

Distributed Roles ü 1 2 3 § 1 . 5 1. 5 0 0 … § 2 0 0. 5 1. 5 0 0 … § 3 0 0 0 position … 0. 5 1. 5 … 1 CAT Role vectors § Words with the same letter in nearby positions will overlap in similarity § E. g. , CAT and ACT 2 3 … A 0 0. 5 1. 5 0 0 … B 0 0 0 … C . 5 1. 5 0 0 … D 0 0 … 3 … What will this type of representation achieve? 0 0 0 1 ACT ü 0 2 A . 5 1. 5 0 0 … B 0 0 0 … C 0 0. 5 1. 5 0 0 … D 0 0 0 … 0 0 0

Similarity of ACT and CAT ü

Similarity of ACT and CAT ü

Distributed Filler ü position E. g. , vowels vs. consonants or phonetic similarity 1

Distributed Filler ü position E. g. , vowels vs. consonants or phonetic similarity 1 BET § A E B D 2 3 … 0. 25. 5. 25 0 0 … A 0 B . 5 1. 5 0 0 … C 0 0 0 … . 25. 5. 25 0 0 … 0. 5 1. 5 0 0 … A 1 . 5 0 0 B 0 0 1 . 5 C 0 0 D 0 0 . 5 1 E. 5 1 0 0 F 0 0 … … … X 0 0 Y 0 0 D Z 0 0 E D E 0 0 0 BAT A 0 B . 5 1. 5 0 0 C 0 0 . 25. 5. 25 0 0 0. 5 1. 5 0 0 … 0. 25. 5. 25 0 0 … 0 0

Representing Magnitudes ü ü E. g. , position along a line Spatial encoding §

Representing Magnitudes ü ü E. g. , position along a line Spatial encoding § also known as place or value encoding § local representation ü Frequency encoding § also known as variable encoding § has properties of a distributed representation

Psychologically Grounded Representations ü ü Music composition network (Mozer, 1994) Task: predict next pitch

Psychologically Grounded Representations ü ü Music composition network (Mozer, 1994) Task: predict next pitch in diatonic scale § with one-hot pitch representation 54. 5% accuracy § with psychologically grounded representation 98. 4% accuracy

Log Scaling ü

Log Scaling ü

Related presentations

Csci 5922Csci 5922
Csci 5922Csci 5922
Deep forest towards an alternative to deep neural networksDeep forest towards an alternative to deep neural networks
Deep neural networks and mixed integer linear optimizationDeep neural networks and mixed integer linear optimization
NvdlaNvdla
Least mean square algorithm in neural networkLeast mean square algorithm in neural network
Neural networks and learning machinesNeural networks and learning machines
Few shot learning with graph neural networksFew shot learning with graph neural networks
Deep learning approach and surface learning approachDeep learning approach and surface learning approach
Visualizing and understanding convolutional neural networksVisualizing and understanding convolutional neural networks
Neural networks for rf and microwave designNeural networks for rf and microwave design
Fuzzy logic lectureFuzzy logic lecture
Meshnet: mesh neural network for 3d shape representationMeshnet: mesh neural network for 3d shape representation
Deep learning vs machine learningDeep learning vs machine learning
Vc dimension neural networkVc dimension neural network
Neural networks ib psychologyNeural networks ib psychology
Audio super resolution using neural networksAudio super resolution using neural networks
Convolutional neural networks for visual recognitionConvolutional neural networks for visual recognition
Image style transfer using convolutional neural networksImage style transfer using convolutional neural networks
Convolutional neural network presentationConvolutional neural network presentation
Pixel recurrent neural networksPixel recurrent neural networks
Matlab u-netMatlab u-net
11-747 neural networks for nlp11-747 neural networks for nlp
Xor problemXor problem
CsrmmCsrmm
On the computational efficiency of training neural networksOn the computational efficiency of training neural networks
Input layerInput layer