Neural Network Background Zhifei Zhang and Hairong Qi

A program that can sense, reason, act, and adapt Algorithms whose performance improve as they are exposed to more data over time • • • 11/23/2020 Naive Bayes k. NN Decision Tree SVM K-Means Dimensionality Reduction, e. g. , FLD, PCA Multi-layer neural networks learn from vast amounts of data 2

11/23/2020 For more detailed history of deep learning https: //www. import. io/post/history-of-deep-learning/ 3

Intuition of Neural Network 11/23/2020 https: //www. quora. com/What-is-an-intuitive-explanation-for-neural-networks 4

Perceptron is usually used to classify the data into two parts. Therefore, it is also known as a Linear Binary Classifier. • Weights shows the strength of the particular node. • A bias value allows you to shift the activation function to the left or right. Bias Weighted Sum 11/23/2020 Activation Function • The activation function map the input between the required values, e. g. , [0, 1]. 5

Perceptron Common activation functions: Bias Weighted Sum 11/23/2020 Activation Function 6

Perceptron Learning Rule Bias Weighted Sum Activation Function Objective: 11/23/2020 Assume 1, directly pass the error backward 7

XOR Problem 11/23/2020 8

Multi-layer Perceptron --- Neural Network in more practice Input layer 11/23/2020 Hidden layer Output layer 9

Multi-layer Perceptron --- Solve the XOR Problem Online demo: https: //lecture-demo. ira. uka. de/neural-network-demo/ 11/23/2020 10

Backpropagation How the multi-layer perceptron (neural network) is learned? Back propagate the error layer-by-layer Output error: 11/23/2020 11

Backpropagation How those multi-layer perceptron (neural network) is learned? Back propagate the error layer-by-layer Output error: 11/23/2020 12

Backpropagation How those multi-layer perceptron (neural network) is learned? Back propagate the error layer-by-layer, and Output error: Back propagated error: Hidden nodes are independent from each other 11/23/2020 13

Convolutional Neural Network (CNN) Handle images which need to preserve the special structure 11/23/2020 14

Convolutional Neural Network (CNN) Le. Net-5 by Yann Le. Cun et al. , 1998 A demo code in Matlab: https: //github. com/ZZUTK/An-Example-of-CNN-on-MNIST-dataset 11/23/2020 15

Convolutional Neural Network (CNN) Appearance of CNN is early (back to 1998) but it booms around 2010 s, WHY? Large dataset (over 10 million) From 2009 11/23/2020 Powerful computing unit From 2010 16

Convolutional Neural Network (CNN) Appearance of CNN is early (back to 1998) but it booms around 2010 s, WHY? Deep convolutional neural networks with competitive performance to human 11/23/2020 Alex. Net, 2012 17

Convolutional Neural Network (CNN) Since 2010 s, the deep convolutional neural networks are mostly referred to as deep learning, and it flourishes in computer vision area until today. 11/23/2020 18

Convolutional Neural Network (CNN) Convolution and Pooling Receptive field 11/23/2020 Where is the activation function? 19

Convolutional Neural Network (CNN) Why deep? What does each layer learn? 11/23/2020 20

Convolutional Neural Network (CNN) Image. Net Large Scale Visual Recognition Challenge (ILSVRC) results show deep learning surpassing human levels of accuracy 11/23/2020 21

Convolutional Neural Network (CNN) How to learn a CNN? Again, Backpropagation. No parameters, do not need update. Directly pass the error backward. 11/23/2020 The same as multi-layer perceptron 22

Convolutional Neural Network (CNN) How to learn a CNN? Again, Backpropagation. Convolutional Neural Network Multi-layer Perceptron 11/23/2020 https: //github. com/ZZUTK/An-Example-of-CNN-on-MNIST-dataset/blob/master/doc/Derivation_of_Backpropagation_in_CNN. pdf 23

11/23/2020 24