Introduction to Deep Learning with Modern Day Applications

Introduction to Deep Learning with Modern Day Applications Presented by Forest Agostinelli

Final Exam • Tue, Dec 11, 4: 00 -6: 00 pm • PSLH-100

Outline • Brief Description • What can deep learning do? • Architectures • Training • Applications • Tips and Tricks • What can deep learning not do…yet?

Brief Description • Input • Processing • Output Processing Connections: “Weights” Input

Brief Description • Input • Processing • Output Processing Nodes: “Neurons” Connections: “Weights” Input

Brief Description Output • Input • Processing • Output Processing block: “Layer” Input

Brief Description • “Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. ” Le. Cun, Y. , Bengio, Y. , & Hinton, G. (2015). Deep learning. nature, 521(7553), 436. Agostinelli, F. , Anderson, M. R. , & Lee, H. (2013). Adaptive multi-column deep neural networks with application to robust image denoising. In NIPS (pp. 1493 -1501). Szegedy, C. , Liu, W. , Jia, Y. , Sermanet, P. , Reed, S. , Anguelov, D. , . . . & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of CVPR (pp. 1 -9). He, K. , Zhang, X. , Ren, S. , & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770 -778).

What can Deep Learning Do? • Classification • Unsupervised Learning • Denoising • Generative models • Games and Puzzles • Natural Language Processing • Sciences

What can Deep Learning Do? Classification • Given an image/sound/etc. determine what class it belongs to Krizhevsky, A. , Sutskever, I. , & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In NIPS (pp. 1097 -1105).

What can Deep Learning Do? Classification • Given an image/sound/etc. determine what class it belongs to Krizhevsky, A. , Sutskever, I. , & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In NIPS (pp. 1097 -1105).

What can Deep Learning Do? Classification • Given an image/sound/etc. determine what class it belongs to Krizhevsky, A. , Sutskever, I. , & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In NIPS (pp. 1097 -1105).

What can Deep Learning Do? Classification • Given an image/sound/etc. determine what class it belongs to Krizhevsky, A. , Sutskever, I. , & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In NIPS (pp. 1097 -1105).

What can Deep Learning Do? Unsupervised Learning • Build meaningful representations of unlabeled data Layer 3 Layer 2 Layer 1 Lee, H. , Grosse, R. , Ranganath, R. , & Ng, A. Y. (2009, June). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26 th annual ICML (pp. 609 -616). ACM.

What can Deep Learning Do? Unsupervised Learning • Build meaningful representations of unlabeled data Layer 3 Layer 2 Layer 1 Lee, H. , Grosse, R. , Ranganath, R. , & Ng, A. Y. (2009, June). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26 th annual ICML (pp. 609 -616). ACM.

What can Deep Learning Do? Denoising • Remove unwanted noise from data Noisy Denoised Agostinelli, F. , Anderson, M. R. , & Lee, H. (2013). Adaptive multi-column deep neural networks with application to robust image denoising. In NIPS (pp. 1493 -1501).

What can Deep Learning Do? Generative Models • “Imagine” new data that is plausibly similar to real-world data Imagination Kingma, D. P. , & Dhariwal, P. (2018). Glow: Generative flow with invertible 1 x 1 convolutions. ar. Xiv preprint ar. Xiv: 1807. 03039.

What can Deep Learning Do? Generative Models • Use imagination to manipulate data Interpolation Kingma, D. P. , & Dhariwal, P. (2018). Glow: Generative flow with invertible 1 x 1 convolutions. ar. Xiv preprint ar. Xiv: 1807. 03039.

What can Deep Learning Do? Generative Models • Use imagination to manipulate data Manipulation Kingma, D. P. , & Dhariwal, P. (2018). Glow: Generative flow with invertible 1 x 1 convolutions. ar. Xiv preprint ar. Xiv: 1807. 03039.

What can Deep Learning Do? Games and Puzzles Silver, D. , Schrittwieser, J. , Simonyan, K. , Antonoglou, I. , Huang, A. , Guez, A. , . . . & Chen, Y. (2017). Mastering the game of Go without human knowledge. Nature, 550(7676), 354. Mc. Aleer, S. *, Agostinelli, F. *, Shmakov, A. *, & Baldi, P. (2018). Solving the Rubik's Cube Without Human Knowledge. ar. Xiv preprint ar. Xiv: 1805. 07470.

What can Deep Learning Do? Natural Language Processing • Understand sentences Sutskever, I. , Vinyals, O. , & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in neural information processing systems (pp. 3104 -3112).

What can Deep Learning Do? Natural Language Processing • Understand sentences Sutskever, I. , Vinyals, O. , & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in neural information processing systems (pp. 3104 -3112).

What can Deep Learning Do? Sciences • Bioinformatics • High Energy Physics • Neuroscience Fortin, N. J. , Elias, G. A. , Li, L. , Agostinelli, F. , Baldi, P. , Shahbaba, B (2018) Sequential Activation of Upcoming Nonspatial Item Representations in Hippocampal Activity. Neuroscience Agostinelli, F. , Ceglia, N. , Shahbaba, B. , Sassone-Corsi, P. , & Baldi, P. (2016). What time is it? Deep learning approaches for circadian rhythms. Bioinformatics, 32(12), i 8 -i 17. Baldi, P. , Sadowski, P. , & Whiteson, D. (2014). Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5, 4308.

Architecture • Layers • Fully Connected • Convolutional • Activation Functions • Outputs

Fully Connected Layers • Connect every processing unit together • Can be used for a plethora of applications Output Input

Convolutional Layers • Convolve filters across the input. • Very effective on structured data such as images, audio, and graphs

Activation Functions • We need the ability to learn non-linear functions of the input • Re. LU and Sigmoid are two of the most popular activations functions. https: //medium. com/@shrutijadon 10104776/survey-on-activation-functions-for-deep-learning-9689331 ba 092

Outputs • Sigmoid • Binary classification • Softmax • Multi-classification • Linear • General purpose regression http: //ufldl. stanford. edu/wiki/index. php/Softmax_Regression

Training • Gradient Descent • Stochastic Gradient Descent • Overfitting/Regularization

Training • Define a loss function • Adjust parameters (weights) to minimize loss • Use gradient descent to adjust parameters Minimized Loss

Stochastic Gradient Descent Data Subset of Data Compute loss Gradient Descent

Stochastic Gradient Descent Data Subset of Data Compute loss Gradient Descent

Stochastic Gradient Descent Data Subset of Data Compute loss Gradient Descent

Stochastic Gradient Descent Data Subset of Data Compute loss Gradient Descent

Overfitting/Regularization • What is overfitting? • Weight Regularization • Dropout

What is overfitting? https: //hackernoon. com/memorizing-is-not-learning-6 -tricks-to-prevent-overfitting-in-machine-learning-820 b 091 dc 42

Weight Regularization • Make the weights less “sensitive” to input Output Input

Dropout • Randomly drop connections between neurons during training Srivastava, N. , Hinton, G. , Krizhevsky, A. , Sutskever, I. , & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929 -1958.

Applications • High Energy Physics • Circadian Rhythms • Neuroscience • Generative Models • Reinforcement Learning

High Energy Physics • Detecting exotic particles (i. e. this Higgs Boson) in collision experiments is a difficult and computationally intensive task. Decision Tree Neural Network Deep Neural Network Baldi, P. , Sadowski, P. , & Whiteson, D. (2014). Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5, 4308.

Circadian Rhythms • Differences in molecular concentration under a control condition and an experimental condition. • These measurements are often carried out at different times. But the times aren’t recorded. • However, differences could be due to the circadian rhythm instead of the experimental condition. Control 04: 00 Control Experimental 12: 00 Experimental Agostinelli, F. , Ceglia, N. , Shahbaba, B. , Sassone-Corsi, P. , & Baldi, P. (2016). What time is it? Deep learning approaches for circadian rhythms. Bioinformatics, 32(12), i 8 -i 17.

Circadian Rhythms • Forces output to lie on a circular manifold. • Captures the cyclic nature of time. • Using a specialized output layer, we can train a network to predict the time of day 0 18 6 12 Agostinelli, F. , Ceglia, N. , Shahbaba, B. , Sassone-Corsi, P. , & Baldi, P. (2016). What time is it? Deep learning approaches for circadian rhythms. Bioinformatics, 32(12), i 8 -i 17.

Circadian Rhythms • The resulting architecture can impute the time an experiment was carried out within 2 hours Agostinelli, F. , Ceglia, N. , Shahbaba, B. , Sassone-Corsi, P. , & Baldi, P. (2016). What time is it? Deep learning approaches for circadian rhythms. Bioinformatics, 32(12), i 8 -i 17.

Unsupervised Learning: Autoencoders • Train a deep neural network to reproduce its input • Use a layer to force it to encode a higher level representation Decoder Bottleneck layer Encoder

Neuroscience • We can then visualize how brain activity changes over time Fortin, N. J. , Elias, G. A. , Li, L. , Agostinelli, F. , Baldi, P. , Shahbaba, B (2018) Sequential Activation of Upcoming Nonspatial Item Representations in Hippocampal Activity. Neuroscience

Generative Models • Sometimes, it is beneficial to have the ability to generate realistic data • There are many methods to this, each with pros and cons • • Variational autoencoders Restricted Boltzmann machines Flow based generative models Generative Adversarial Networks

Generative Models: Generative Adversarial Networks Goodfellow, I. , Pouget-Abadie, J. , Mirza, M. , Xu, B. , Warde-Farley, D. , Ozair, S. , . . . & Bengio, Y. (2014). Generative adversarial nets. In NIPS (pp. 2672 -2680). http: //blog. aylien. com/introduction-generative-adversarial-networks-code-tensorflow/

Generative Models: Generative Adversarial Networks J. -Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. ar. Xiv preprint ar. Xiv: 1703. 10593, 2017.

Generative Models: Generative Adversarial Networks J. -Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. ar. Xiv preprint ar. Xiv: 1703. 10593, 2017.

Generative Models: Generative Adversarial Networks J. -Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. ar. Xiv preprint ar. Xiv: 1703. 10593, 2017.

Generative Models: Generative Adversarial Networks J. -Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. ar. Xiv preprint ar. Xiv: 1703. 10593, 2017.

Generative Models: Generative Adversarial Networks • Generative adversarial networks have been extended to a plethora of tasks • In addition to images, synthetic audio has been created that closely mimics real people J. -Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. ar. Xiv preprint ar. Xiv: 1703. 10593, 2017.

Reinforcement Learning Agostinelli, F. , Hocquet, G. , Singh, S. , & Baldi, P. (2018). From Reinforcement Learning to Deep Reinforcement Learning: An Overview. In Braverman Readings in Machine Learning. Key Ideas from Inception to Current State (pp. 298 -328). Springer, Cham.

Reinforcement Learning •

Reinforcement Learning • All grey boxes can be replaced by a deep neural network Agostinelli, F. , Hocquet, G. , Singh, S. , & Baldi, P. (2018). From Reinforcement Learning to Deep Reinforcement Learning: An Overview. In Braverman Readings in Machine Learning. Key Ideas from Inception to Current State (pp. 298 -328). Springer, Cham.

Reinforcement Learning Mc. Aleer, S. , Agostinelli, F. , Shmakov, A. , & Baldi, P. (2018). Solving the Rubik's Cube Without Human Knowledge. ar. Xiv preprint ar. Xiv: 1805. 07470.

Tips and Tricks • Input Normalization • Learned Activation Functions • Residual Learning

Input Normalization • Deep neural networks are math • Data normalization is crucial to ensuring numerical stability

Learned Activation Functions Agostinelli, F. , Hoffman, M. , Sadowski, P. , & Baldi, P. (2014). Learning activation functions to improve deep neural networks. ar. Xiv preprint ar. Xiv: 1412. 6830. He, K. , Zhang, X. , Ren, S. , & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (pp. 1026 -1034).

Residual Learning • Despite many advances, deep neural networks are still hard to train • Training can become more difficult as the number of layers increases • Adding skip connections allows us to train networks with hundreds of layers He, K. , Zhang, X. , Ren, S. , & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770 -778).

What Can Deep Learning Not Do…Yet? • Math

What Can Deep Learning Not Do…Yet? • Math • Logic https: //en. wikipedia. org/wiki/Logic_(musician)

What Can Deep Learning Not Do…Yet? • Math • Logic • Explain Itself

What Can Deep Learning Not Do…Yet? • Math • Logic • Explain Itself • Theoretical Guarantees

What Can Deep Learning Not Do…Yet? • Math • Logic • Explain Itself • Theoretical Guarantees • Safety Guarantees This Photo by Unknown Author is licensed under CC BY-ND

Questions?