NLP Deep Learning Long ShortTerm Memory Networks LSTM

  • Slides: 21
Download presentation
NLP

NLP

Deep Learning Long Short-Term Memory Networks (LSTM)

Deep Learning Long Short-Term Memory Networks (LSTM)

LSTM Motivation Remember how we update an RNN? y 3 Cost softmax wy h

LSTM Motivation Remember how we update an RNN? y 3 Cost softmax wy h 0 wh σ wx h 1 wh σ wx h 2 wh σ h 3 wx x 1 x 2 x 3 The cat sat [slides from Catherine Finegan-Dollak]

The Vanishing Gradient Problem • Deep neural networks use backpropagation. • Back propagation uses

The Vanishing Gradient Problem • Deep neural networks use backpropagation. • Back propagation uses the chain rule. • The chain rule multiplies derivatives. • Often these derivatives between 0 and 1. • As the chain gets longer, products get smaller • until they disappear. Wolfram|Alpha Derivative of sigmoid function

Or do they explode? • With gradients larger than 1, • you encounter the

Or do they explode? • With gradients larger than 1, • you encounter the opposite problem • with products becoming larger and larger • as the chain becomes longer and longer, • causing overlarge updates to parameters. • This is the exploding gradient problem.

Vanishing/Exploding Gradients Are Bad. • If we cannot backpropagate very far through the network,

Vanishing/Exploding Gradients Are Bad. • If we cannot backpropagate very far through the network, the network cannot learn long-term dependencies. • My dog [chase/chases] squirrels. vs. • My dog, whom I adopted in 2009, [chase/chases] squirrels. 6

LSTM Solution § Use memory cell to store information at each time step. §

LSTM Solution § Use memory cell to store information at each time step. § Use “gates” to control the flow of information through the network. § Input gate: protect the current step from irrelevant inputs § Output gate: prevent the current step from passing irrelevant outputs to later steps § Forget gate: limit information passed from one cell to the next

Transforming RNN to LSTM h 0 wh σ wx x 1 u 1

Transforming RNN to LSTM h 0 wh σ wx x 1 u 1

Transforming RNN to LSTM c 0 h 0 wh σ wx x 1 u

Transforming RNN to LSTM c 0 h 0 wh σ wx x 1 u 1

Transforming RNN to LSTM f 1 c 0 h 0 wh σ wx x

Transforming RNN to LSTM f 1 c 0 h 0 wh σ wx x 1 u 1 i 1 + c 1

Transforming RNN to LSTM f 1 c 0 h 0 wh σ wx x

Transforming RNN to LSTM f 1 c 0 h 0 wh σ wx x 1 u 1 i 1 + c 1

Transforming RNN to LSTM f 1 c 0 h 0 wh σ wx x

Transforming RNN to LSTM f 1 c 0 h 0 wh σ wx x 1 u 1 i 1 + c 1

Transforming RNN to LSTM f 1 c 0 whf h 0 wh σ wx

Transforming RNN to LSTM f 1 c 0 whf h 0 wh σ wx x 1 σ u 1 wxf i 1 + c 1

Transforming RNN to LSTM f 1 c 0 whi h 0 wh σ wx

Transforming RNN to LSTM f 1 c 0 whi h 0 wh σ wx x 1 σ u 1 wxi i 1 + c 1

Transforming RNN to LSTM f 1 c 0 + c 1 tanh h 0

Transforming RNN to LSTM f 1 c 0 + c 1 tanh h 0 wh σ wx x 1 u 1 i 1 o 1 h 1

LSTM for Sequences f 1 c 0 + f 2 c 1 tanh h

LSTM for Sequences f 1 c 0 + f 2 c 1 tanh h 0 wh σ wx u 1 i 1 o 1 + f 2 c 2 wh σ wx x 1 x 2 The cat u 2 i 2 o 2 c 2 tanh h 1 + h 2 wh σ wx x 2 sat u 2 i 2 o 2 h 2

LSTM Applications http: //www. cs. toronto. edu/~graves/handwriting. html • • Language identification (Gonzalez-Dominguez et

LSTM Applications http: //www. cs. toronto. edu/~graves/handwriting. html • • Language identification (Gonzalez-Dominguez et al. , 2014) Paraphrase detection (Cheng & Kartsaklis, 2015) Speech recognition (Graves, Abdel-Rahman, & Hinton, 2013) Handwriting recognition (Graves & Schmidhuber, 2009) Music composition (Eck & Schmidhuber, 2002) and lyric generation (Potash, Romanov, & Rumshisky, 2015) Robot control (Mayer et al. , 2008) Natural language generation (Wen et al. 2015) (best paper at EMNLP) Named entity recognition (Hammerton, 2003)

Related Architectures: GRU h 0 wh z 1 r 1 σ wx x 1

Related Architectures: GRU h 0 wh z 1 r 1 σ wx x 1 + ĥ 1 h 1 1 -z 1 Chung et al. (2014) reports comparable performance to LSTM

Related Architectures: Tree LSTMs f c 0 h 0 x 2 c 1 u

Related Architectures: Tree LSTMs f c 0 h 0 x 2 c 1 u 2 i c 2 o h 2 h 1 f Tai, Socher, Manning 2015

External Links • http: //colah. github. io/posts/2015 -08 -Understanding. LSTMs/

External Links • http: //colah. github. io/posts/2015 -08 -Understanding. LSTMs/

NLP

NLP

NLP Introduction to NLP Text Generation Basic NLPNLP Introduction to NLP Text Generation Basic NLP
Deep Levels Deep Levels Deep Centers Deep TrapsDeep Levels Deep Levels Deep Centers Deep Traps
EXPLOITING LSTM STRUCTURE IN DEEP NEURAL NETWORKS FOREXPLOITING LSTM STRUCTURE IN DEEP NEURAL NETWORKS FOR
LSTM Long Short Term Memory Image Processing SeminarLSTM Long Short Term Memory Image Processing Seminar
LSTM Long Short Term Memory The basic structureLSTM Long Short Term Memory The basic structure
ShortTerm Memory Learning Objectives Understand what memory isShortTerm Memory Learning Objectives Understand what memory is
Deep neural networks DNNs Conventional and deep networksDeep neural networks DNNs Conventional and deep networks
HUMAN MEMORY Sensory memory Short term memory LongHUMAN MEMORY Sensory memory Short term memory Long
Longterm Memory Explicit memory Episodic Memory Semantic MemoryLongterm Memory Explicit memory Episodic Memory Semantic Memory
A Hybrid Residual Network and Long ShortTerm MemoryA Hybrid Residual Network and Long ShortTerm Memory
EEG Classification Using Long ShortTerm Memory Recurrent NeuralEEG Classification Using Long ShortTerm Memory Recurrent Neural
CSCI 5922 Neural Networks and Deep Learning DeepCSCI 5922 Neural Networks and Deep Learning Deep
CSCI 5922 Neural Networks and Deep Learning DeepCSCI 5922 Neural Networks and Deep Learning Deep
Deep asleep deep asleep Deep asleep it liesDeep asleep deep asleep Deep asleep it lies
Shortterm working memory Students of memory e gShortterm working memory Students of memory e g
Shortterm and Working Memory 1 Definition of memoryShortterm and Working Memory 1 Definition of memory
Shortterm working memory Students of memory e gShortterm working memory Students of memory e g
Working Memory Active shortterm memory Maintenance of taskrelevantWorking Memory Active shortterm memory Maintenance of taskrelevant
LSTM Recurrent Neural Networks Brandeis CS 114 SpringLSTM Recurrent Neural Networks Brandeis CS 114 Spring
Recurrent Neural Networks LSTM Advisor S J WangRecurrent Neural Networks LSTM Advisor S J Wang
SHORTTERM RENTAL ORDINANCE Shortterm Rental Rent or leaseSHORTTERM RENTAL ORDINANCE Shortterm Rental Rent or lease
ShortTerm LongTerm Project Status ShortTerm Projects Madison CountyShortTerm LongTerm Project Status ShortTerm Projects Madison County
Chapter 20 ShortTerm Financial Planning Chapter 20 ShortTermChapter 20 ShortTerm Financial Planning Chapter 20 ShortTerm
Deep Learning Hungyi Lee Deep learning attracts lotsDeep Learning Hungyi Lee Deep learning attracts lots
First Steps With Deep Learning Course Deep learningFirst Steps With Deep Learning Course Deep learning
Deep Learning Early Work l Why Deep LearningDeep Learning Early Work l Why Deep Learning
The tsunami of Deep Learning over NLP GiuseppeThe tsunami of Deep Learning over NLP Giuseppe
NLP Deep Learning Sequencetosequence and Neural Machine TranslationNLP Deep Learning Sequencetosequence and Neural Machine Translation
NLP Master Coach Skab enestende forandringer NLP MasterNLP Master Coach Skab enestende forandringer NLP Master
UCSF NLP Infrastructure May 2018 NLP Infrastructure AttributesUCSF NLP Infrastructure May 2018 NLP Infrastructure Attributes
NLP Introduction to NLP Information Extraction Information ExtractionNLP Introduction to NLP Information Extraction Information Extraction
NLP Introduction to NLP Sentence Simplification Sentence SimplificationNLP Introduction to NLP Sentence Simplification Sentence Simplification
NLP Introduction to NLP Classic parsing methods SNLP Introduction to NLP Classic parsing methods S
NLP Introduction to NLP Syntax Syntax Is languageNLP Introduction to NLP Syntax Syntax Is language
NLP Introduction to NLP Word Distributions Word DistributionsNLP Introduction to NLP Word Distributions Word Distributions
Statistical NLP Lecture 2 Introduction to Statistical NLPStatistical NLP Lecture 2 Introduction to Statistical NLP
NLP Introduction to NLP Semantic Role Labeling SyntacticNLP Introduction to NLP Semantic Role Labeling Syntactic
Clinical NLP in North Germanic Languages Clinical NLPClinical NLP in North Germanic Languages Clinical NLP
NLP Introduction to NLP Language Models Probabilistic LanguageNLP Introduction to NLP Language Models Probabilistic Language
NLP Introduction to NLP NLG Systems FOG FOGNLP Introduction to NLP NLG Systems FOG FOG