Introduction to RNN and LSTM PRESENTOR WENGUANG MAO

Introduction to RNN and LSTM PRESENTOR: WENGUANG MAO

Traditional Neural Network q Input Layer q Hidden Layers q Output Layer q Activation

Sequence Problems q The output depends on q Current Input q Previous Output q Traditional neural network cannot capture such relationship

Recurrent Neural Network USE THE PREVIOUS OUTPUT AS THE INPUT OF CURRENT NETWORK

Different Configurations

Problem of RNN q The Concept of RNN is great q Having a practical problem for training RNN

Gradient Vanishing

The Values of Derivative q Depending on the activation functions

What’s the results q The partial derivative is close to zero q The output is not related to far pervious inputs q Long dependency is vanishing

Solutions q Change activation functions q The problem can be alleviated somewhat but not eliminated q May affect the learning performance q LSTM

Long Short-Term Memory KEY INSIGHTS: BUILD A DIRECT PATH FROM PREVIOUS INPUTS TO THE CURRENT OUTPUT

LSTM Breakdown q Hidden states q Input gates q Forget gates q Output gates

Hidden States q Carry previous information q Build a direct path

Gates

Input Gates WHAT IS KEPT F ROM PREVIOUS STAT ES

Forget Gate WHAT IS ADDED T O T HE HIDDEN S TATE

Output Gates WHAT IS REPORTED AS OUT PUT

LSTM Variants

1 2 RNN is used for sequence / time-series problem Gradient vanishing problem limits effectiveness 3 LSTM cell to eliminate problem Conclusion 4 Practically LSTM works much better than RNN