Deep Learning Authors Yann Le Cun Yoshua Bengio

Deep Learning Authors: Yann Le. Cun, Yoshua Bengio, Geoffrey Hinton Presented by Nick Harris

Outline ● ● ● ● The Success of Deep Learning The Basics of Neural Networks Forward and Back propagation Supervised Learning Convolutional Neural Networks Recurrent Neural Networks The Future of Deep Learning

Success of Deep Learning ● ● Deep Learning has become increasingly popular in recent years It is increasingly used in consumer products such as smartphones, and in commercial websites It has also greatly improved state-of-the-art performance in a wide variety of tasks such as speech and object recognition What makes Deep Learning so successful?

Success (continued) ● Deep Learning offers numerous advantages over older conventional machine learning techniques: ○ ○ ○ ● Deep Learning techniques have a greater capacity to process raw data Conventional techniques often require a lot of domain expertise and precise engineering for good pattern recognition. NNs on the other hand can perform well with little prior knowledge. Deep Learning techniques tend to continue to improve when given vastly larger amounts of data Two Deep Learning models that have been especially successful are the Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN)

Neural Networks Neural Network systems consist of large networks of artificial neurons ● ● ● These neurons map a set of inputs X to an output y Parameters: weights (w), bias (b) Activation Function (phi) to introduce nonlinearity

Neural Networks (continued) Some popular activation functions:

Neural Networks (continued) ● ● Many neurons are combined to form a network. Maps an arbitrary number of inputs X to and arbitrary number of outputs Y A Neural Network with even one hidden layer is sufficient to compute any function Deep Learning is where many successive layers are used. ○ ○ These many layers learn increasingly abstract representation of the data Learn much better than small network in practice

Forward/Back Propagation ● ● Forward Propagation is the process of activating the network with to produce an output Back Propagation is the process of learning by adjusting the weights of the network

Gradient Descent ● ● ● Learning can be thought of as finding an optimal point in a high dimensional landscape Computing the error gradient and adjusting the weights can be thought of as “hillclimbing” where we move progressively toward the optimal point Stochastic Gradient Descent is a technique used in practice that speeds up learning by computing the average gradient over a random set of samples (but introduces noise).

Supervised Learning ● ● ● One of the most common learning processes for neural networks Asks the network to make a prediction, then use the correct answer to adjust the weights with back propagation Requires large data sets

Convolutional Neural Nets (CNNs) ● A special class of neural networks that excel at processing images ● Uses Convolutional and Pooling layers to break down images

CNNs (continued) Convolutional Layer Pooling Layer

Value of CNNs have proven to be extremely good at image classification and object detection. As a result they are widely used in the computer vision field.

Recurrent Neural Networks (RNNs) ● ● RNNs are a class of neural networks that excel at processing sequential data (sensitive to time and order) One especially successful application is natural language processing This includes predicting the next character of text, or the next word in a sequence A more sophisticated application is translation by mapping text to a “thought vector”, then mapping to text in another language

RNNs (continued) ● ● RNNs process input in a sequence one step at a time, receiving inputs from previous time steps Maintain a hidden “state vector” that contains information about the history of past elements in the sequence RNNs were initially problematic to train, but recent advances in architecture and training have allowed them to produce state-of-the-art results Can be augmented with a more explicit memory to form an Long Short Term Memory (LSTM) network, which has proven to be more powerful

Future of Deep Learning ● ● ● Unsupervised Learning is not as popular as Supervised of late, but is considered to have great potential Interest in combining CNNs and RNNs with reinforcement learning for difficult computer vision tasks Interest in improving natural language processing performance

Questions?