CSC 578 Neural Networks and Deep Learning 5

CSC 578 Neural Networks and Deep Learning 5. Convolutional Neural Networks (1) (Some figures adapted from NNDL book) Noriko Tomuro 1

1 Convolutional Neural Networks • Convolutional Neural Networks (CNNs) are a variation of a multilayer neural network, typically used for recognizing/classifying 2 D images. • A CNN consists of an input and an output layer, as well as multiple hidden layers. The hidden layers of a CNN typically consist of convolutional layers, pooling layers, and fully connected layers. Noriko Tomuro 2

• Convolutional layers apply a convolution operation to the input. The operation applies a filter function/kernel on a receptive field/window of some size over the input data. • A receptive field is moved/slid over the input, stopping at every pixel or skipping over a fixed number of pixels (stride). • You can apply as many different filter functions, where each function creates a convolution feature map. Noriko Tomuro 3

• Next, for each convolution feature map, a pooling operation is applied which combines a cluster of convolution features to a single value. • Common functions are max pooling and average pooling. • Typically a CNN consists of several convolution and pooling layers. Noriko Tomuro 4

• Then the last pooling layer is flattened to a 1 D vector (possibly after dropping some nodes), which gets connected a network of fully connected (FC) layers. • The final output layer is in the same principle as the usual multilayer neural network. Noriko Tomuro 5

• Essentially CNNs extract features in the input images hierarchically: – Early layers extract low-level features such as lines, edges and curves. – Middle layers combine low-level features to extract mid-level features such as texture and parts of objects. – Later layers combine mid-level features to extract high-level (semantic) features such as full objects. Noriko Tomuro 6

• CNN learning is to learn kernels, which are weights in the filters. • As with feed-forward networks, we start with random small weights and adjust them as we go through training epochs. • Example of learned kernels: Noriko Tomuro 7

Examples of CNN Applications • CNNs have been applied to many tasks, primarily for images (2 -D with various number of channels) including videos. • Example: Image Classification CIFAR-10 Dataset Noriko Tomuro 8

• Example: Object Detection and Segmentation. Noriko Tomuro 9

• Example: Autonomous Driving • Note: CNNs for 1 -D (e. g. texts) or 3 -D (e. g. videos) data have also been developed and commonly used these days. Noriko Tomuro 10

2 Convolution Kernel Noriko Tomuro 11

Noriko Tomuro 12

• Sometimes an activation function (applied after kernel) is considered a separate layer. Alex. Net Noriko Tomuro 13

3 Shared Weights and Biases • Nodes in a receptive field to a node on the convolution layer are connected with weights. There is also a bias. Those weights and bias are shared – same values are used for a given filter as a receptive field is moved on the same (input or intermediate convolutional) layer. • For example, the output of the jkth convolution node from a 5 x 5 receptive field would be Noriko Tomuro And those weights are learned by training. 14

• By sharing the same weights (and bias) of the same filter, all the neurons in a convolution layer detect exactly the same feature. • This makes a filter “shift invariant” – being able to find the feature anywhere in the entire image (or the preceding layer) wherever it occurred. • For example, filters could detect edges, lines, corners and blobs of color. • Animation of sliding window of receptive field, max pool etc. Noriko Tomuro 15

• Example: Applying a learned filter that detects a diagonal line (in yellow box) over the input image creates a feature map in which all diagonal lines in the input image were identified. Noriko Tomuro 16

4 Pooling • A pooling layer takes each feature map output from the convolutional layer and prepares a condensed feature map. • It is essentially to down-sample the feature map. Noriko Tomuro 17

• Two common pooling methods: max and average pooling. • Pooling helps to reduce overfitting by providing an abstracted form of the representation. • It also reduces the computational cost by reducing the number of parameters to learn. Noriko Tomuro 18

5 CNN Learning • CNNs are a variation of feed-forward deep neural network. So all of the concepts we learned so far on neural networks apply, such as activation functions and cost/loss functions. • CNN learning is to learn the weights between layers. • But the weights between a hidden/convolution layer and its preceding layer are a kernel (modulo activation function). So essentially CNN learning is to learn convolution kernels. • The learning algorithm is basically the same from what we’ve learned, by backpropagation using gradient descent. Noriko Tomuro 19

• Cross-correlation I K Convolution Noriko Tomuro K I 20

Backprop in CNN: (1) Forward Propagation Noriko Tomuro 21

Convolution as Sparse Connection

Convolution as a Matrix Operation If the input and output were to be unrolled into vectors from left to right, top to bottom, the convolution could be represented as a sparse matrix efficient implementation!! Reference: Theano documentation

Backprop in CNN: (2) Backward Propagation where which is Noriko Tomuro 25

So we get Noriko Tomuro 26

Noriko Tomuro 27

Noriko Tomuro 28

Noriko Tomuro 29

Noriko Tomuro 30

● References: ○ http: //courses. cs. tau. ac. il/Caffe_workshop/Bootcamp/pd f_lectures/Lecture%203%20 CNN%20%20 backpropagation. pdf ○ https: //medium. com/@14 prakash/back-propagation-isvery-simple-who-made-it-complicated-97 b 794 c 97 e 5 c ○ https: //www. jefkine. com/general/2016/09/05/backpropa gation-in-convolutional-neural-networks/ Noriko Tomuro 31