Deep Convolutional Neural Networks for Image Processing An

Deep Convolutional Neural Networks for Image Processing An Overview on Convolutional Neural Networks for Image Classification and Image Segmentation Delft University of Technology Franziska Riegger January 29, 2019

Deep Convolutional Neural Networks for Image Processing: extracting useful information from an image https: //www. researchgate. net/figure/An-overview-image-of-a-dendritic-microstructure-turbine-blade-made-from-Ni-base_fig 1_278327694 Franziska Riegger Conv. Nets for Image Option A: safety critical Option B: shorten maintenance interval Option C: no consequences 29 January 2019 2/35

Deep Convolutional Neural Networks for Image Processing Image Classification assign image to one class Image Segmentation assign each pixel to one class Deep Convolutional Networks (Conv. Nets) with software library Tensorflow Franziska Riegger Conv. Nets for Image 29 January 2019 3/35

Outline 1. Principles of Machine Learning 2. Image Classification with Conv. Nets 3. Image Segmentation with Conv. Nets Franziska Riegger Conv. Nets for Image 29 January 2019 4/35

Principles of Machine Learning 1. Fully-connected Neural Networks 2. Improvements Franziska Riegger Conv. Nets for Image 29 January 2019 5/35

Neural Networks Learning Problem Model class Performance Measure Training Validation Franziska Riegger Conv. Nets for Image 29 January 2019 6/35

Neural Networks – Learning Problem: classify digit in an image Franziska Riegger Conv. Nets for Image 29 January 2019 7/35

Neural Networks – Model Class Fully-connected neural network : neuron Franziska Riegger Conv. Nets for Image 29 January 2019 8/35

Neural Networks – Performance Measure Performance measure: Optimal model: Franziska Riegger classification or generalization error Conv. Nets for Image 29 January 2019 9/35

Neural Networks – Empirical Risk Minimization How to optimize for any image if only limited data is available? ? Empirical Risk Minimization (ERM) Franziska Riegger Conv. Nets for Image 29 January 2019 10/35

Neural Networks – Training: Franziska Riegger Conv. Nets for Image 29 January 2019 11/35

Neural Networks – Validation: overfitting Franziska Riegger Conv. Nets for Image 29 January 2019 12/35

Principles of Machine Learning 1. Fully-connected Neural Networks 2. Improvements Franziska Riegger Conv. Nets for Image 29 January 2019 13/35

Neural Networks – Troubleshooting Overfitting Prevention: adapt model to learning problem Learning Problem: feature extraction feature: useful information in data e. g. eye color Franziska Riegger Conv. Nets for Image 29 January 2019 14/35

Neural Networks – Sparse Connectivity Status quo network: Franziska Riegger parameter matrix extracts features in entire image Conv. Nets for Image 29 January 2019 15/35

Neural Networks – Sparse Connectivity Status quo network: parameter matrix extracts features in entire image Improvement 1: cover only relevant area reduce connectivity of parameter matrix Discrete convolution with kernel results in feature map Franziska Riegger Conv. Nets for Image 29 January 2019 16/35

Neural Networks – Parameter Sharing Status quo network: one set of parameter for all features Improvement 2: one parameter set for each feature Franziska Riegger Conv. Nets for Image 29 January 2019 17/35

Neural Networks – Parameter Sharing Status quo network: one set of parameter for all features Improvement 2: one parameter set for each feature results in several feature maps: matrix tensor Franziska Riegger Conv. Nets for Image 29 January 2019 18/35

Neural Networks – Convolutional Networks Learning Problem: feature extraction layers with multiple discrete convolution instead of one common matrix multiplication convolutional layer fundament of convolutional network Franziska Riegger Conv. Nets for Image 29 January 2019 19/35

Image Classification with Conv. Nets Franziska Riegger Conv. Nets for Image 29 January 2019 20/35

Image Classification – Conv. Nets Simple classifier: too simple for images fully-connected network input image classifications classification Franziska Riegger Conv. Nets for Image 29 January 2019 21/35

Image Classification – Conv. Nets Simple classifier: fully-connected network too simple for images feed classifier only with input relevant information image classifications additional feature extraction Franziska Riegger Conv. Nets for Image classification 29 January 2019 22/35

Image Classification – Feature Extraction Complex concepts How do we know that this is a car? wheels, headlights, … How do we know that this is a wheel? Simple concepts horizontal, vertical lines Conv. Nets: hierarchical feature extraction Franziska Riegger Conv. Nets for Image 29 January 2019 23/35

Image Classification – Hierarchical Feature Extraction Step 1: Simple feature convolutional layer Step 2: merge to more complex concept translation invariance downsampling layer Franziska Riegger Conv. Nets for Image 29 January 2019 24/35

Image Classification – Downsampling MAX-Pooling: example of downsampling grid-wise application of MAX function MAX Input feature map y x 1 1 2 4 5 6 7 8 6 8 3 2 1 0 3 4 1 2 3 4 pooled feature map invariance towards translation and reduces feature map resolution Franziska Riegger Conv. Nets for Image 29 January 2019 25/35

Image Classification – Architecture Feature extraction: stackwise convolutional and downsampling stage Classification: fully-connected network convolutional layer downsampling layer input image classifications feature classification extraction Franziska Riegger Conv. Nets for Image 29 January 2019 26/35

Image Classification– The Digits Validation Franziska Riegger Conv. Nets for Image 29 January 2019 27/35

Image Classification – Digits Franziska Riegger Conv. Nets for Image 29 January 2019 28/35

Image Classification – Digits Franziska Riegger Conv. Nets for Image 29 January 2019 29/35

Image Segmentation with Conv. Nets Franziska Riegger Conv. Nets for Image 29 January 2019 30/35

Image Classification vs Segmentation Image Classification Image Segmentation What? Where ? airplanes image-wise classification feature extraction based on translation invariance Franziska Riegger Conv. Nets for Image pixel-wise classification feature extraction with location 29 January 2019 31/35

Image Segmentation – Output size Classification Conv. Nets maps feature map to fixed size prediction vector Segmentation Conv. Nets maps feature map to prediction map of same size convolutional layer: 1 x 1 kernel fully-connected layer Franziska Riegger Conv. Nets for Image 29 January 2019 32/35

Image Segmentation – Output size Classification Conv. Nets maps feature map to fixed size prediction vector Segmentation Conv. Nets maps feature map to prediction map of same size convolutional layer: 1 x 1 kernel fully-connected layer Franziska Riegger Conv. Nets for Image 29 January 2019 33/35

Image Segmentation – Upsampling input space downsampling feature space discard locality information and decrease resolution Franziska Riegger Conv. Nets for Image 29 January 2019 34/35

Image Segmentation – Upsampling input space downsampling feature space discard locality information and decrease resolution Franziska Riegger Conv. Nets for Image upsampling input spac e add locality information and increase resolution 29 January 2019 35/35

Image Segmentation – Upsampling Unpooling: example of upsampling reverses MAX-pooling Switch variable (x, y) y x (2, 1) (1, 2) (2, 1) 1 1 2 4 5 6 7 8 3 2 1 0 6 1 2 3 4 3 Franziska Riegger pooling 0 0 0 6 0 8 8 3 0 0 0 4 0 0 3 0 Conv. Nets for Image unpooling 29 January 2019 36/35

Image Segmentation – Upsampling Unpooling: Franziska Riegger locality information via skip connection Conv. Nets for Image 29 January 2019 37/35

Image Segmentation – Upsampling Unpooling: upsampled matrix is sparse densify by transpose convolution Franziska Riegger Conv. Nets for Image 29 January 2019 38/35

Image Segmentation – Architecture transpose convolutional layer Franziska Riegger Conv. Nets for Image upsampling layer 29 January 2019 39/35

Summary Image Classification two parts: feature extraction and classification feature extraction based on translation invariance Image Segmentation inherent tension between feature extraction and pixel-wise classification redesign Conv. Nets for Classification In general Tailoring network to learning problem improves performance Franziska Riegger Conv. Nets for Image 29 January 2019 40/35

What’s coming next? Is image segmentation based on Deep Learning applicable to determine different phenotypical structures of a material with nearly human-like precision? Franziska Riegger Conv. Nets for Image 29 January 2019 41/35