RNN and LSTM Using MXNet Cyrus M Vahid

RNN and LSTM Using MXNet Cyrus M Vahid, Principal Solutions Architect cyrusmv@amazon. com May 2017 © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Sequences and Long-Term Dependency • The Spanish King officially abdicated in … of his …, Felipe will be confirmed tomorrow as the new Spanish ….

Sequences and Long-Term Dependency • The Spanish King officially abdicated in favour of his son , Felipe will be confirmed tomorrow as the new Spanish King.

Recurrent Neural Networks Image Labeling Image Caption Sentiment Analysis Machine Translation Video Labeling

Computational Graphs sum x z x sqr dot x y w x

Recurrent Neural Networks – Training

Recurrent Neural Networks – Training Unfold

Recurrent Neural Networks – Training Unfold

Recurrent Neural Networks – Training in Time Unfold

Recurrent Neural Networks – Summarizing

Recurrent Neural Networks – Expanding

Encode-Decoder (seq 2 seq)

Challenge of Long-Term Dependencies • • Deep RNNs have a deep computational graph They apply the same operation at each time-step for a long temporal sequence. • More specially in RNNs

LSTM – Internal Structure Output gate (sigmoid) controls the output Linear self-loop Forget gate (sigmoid) controls The value of state self-loop Input gate (non-linear) controls the input

LSTM – The Math

Back Propagation Through a Memory Cell (from Hinton Lecture)

LSTM – The Solution • • The LSTM Architecture consists of a set of recurrently connected subnets, known as memory blocks An LSTM network is structures similar to a normal RNN, except that non-linear units in the hidden layers are replaced by memory blocks. Hidden layers can be connected to any form of output layer. Each block has three gates, input, output, and forget. This allows for protecting the information inside and dynamically changing the values inside.

Handwriting Recognition by Alex Graves

RNN In MXNet

RNN In MXNet

RNN In MXNet

References • How ANN predicts Stock Market Indices? by Vidya Sagar Reddy Gopala, Deepak Ramu • Deep Learning, O’Reilly Media Inc. ISBN: 9781491914250; By: Josh Patterson and Adam Gibson • Artificial Intelligence for Humans, Volume 3: Deep Learning and Neural Networks, Create. Space Independent Publishing Platform; ISBN: 1505714346; By: Jeff Heaton • Deeplearningbook. org and Deep Learning from MIT Press. ISBN: 0262035618 • Supervised Sequence Labling with Recurrent Neural Networks; Alex Grave. http: //www. springer. com/de/book/9783642247965 or https: //www. cs. toronto. edu/~graves/preprint. pdf • Coursera; Neural Networks for Machine Learning by Jeoff Hinton at University of Toronto • https: //www. researchgate. net/figure/267214531_fig 1_Figure-1 -RNN-unfolded-in-time. Adapted-from-Sutskever-2013 -with-permission • www. yann-lecun. com • http: //www. wildml. com/2015/09/recurrent-neural-networks-tutorial-part-1 -introduction-tornns/ • https: //github. com/cyrusmvahid/mxnet-notebooks/tree/master/python

Thank You Cyrus M Vahid cyrusmv@amazon. com