Chair for Computer Aided Medical Procedures Augmented Reality

Chair for Computer Aided Medical Procedures & Augmented Reality Master Seminar: Deep Learning for Medical Applications Fickle. Net: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference Student: Till Richter, till. richter@tum. de Tutor: Tariq Mousa Bdair, t. bdair@tum. de

Outline 1. Introduction 2. Related Work 3. Methodology 4. Experiment 5. Conclusion and Outlook Computer Aided Medical Procedures October 17, 2021 Slide 2

Introduction Motivation • Pixel-accurate Segmentation • Efficient Computability • Limited Data Computer Aided Medical Procedures October 17, 2021 Slide 3

Introduction Problem Statement Pixel-accurate Segmentation increases Performance of Tumor Segmentation. However, obtaining Labels is expensive. Fickle. Net provides a Solution using Weakly Supervised Learning. Computer Aided Medical Procedures October 17, 2021 Slide 4

Related Work Overview • Principles: Class Activation Map [1] • Considering Pixel Information: Image-level [2 -4] • Considering Feature Information: Feature-level [5, 6] • Random Walk based: Region growing [7 -9] [1]: B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization [2]: K. Li, Z. Wu, K. -C. Peng, J. Ernst, and Y. Fu. Tell me where to look: Guided attention inference network [3]: K. K. Singh and Y. J. Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization [4]: Y. Wei, J. Feng, X. Liang, M. -M. Cheng, Y. Zhao, and S. Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach [5]: X. Zhang, Y. Wei, J. Feng, Y. Yang, and T. Huang. Adversarial complementary learning for weakly supervised object Localization [6]: D. Kim, D. Cho, D. Yoo, and I. S. Kweon. Two-phase learning for weakly supervised object localization [7]: R. Fan, Q. Hou, M. -M. Cheng, T. -J. Mu, and S. -M. Hu. s 4 net: Single stage salient-instance segmentation. [8]: A. Kolesnikov and C. H. Lampert. Seed, expand constrain: Three principles for weakly-supervised image segmentation [9]: Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly supervised semantic segmentation network with deep seeded region growing Computer Aided Medical Procedures October 17, 2021 Slide 5

Related Work Class Activation Map [1] Goal: Contribution of each hidden Unit in a Neural Net to the Classification Score Drawback: Focus on small discriminative Region of the Object Not suited for semantic Segmentation Image: [1]: B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization Computer Aided Medical Procedures October 17, 2021 Slide 6

Related Work Image Level [2 -4] • Hide Parts of the Image • Drawbacks in Accuracy and Computation Feature Level • Hide Parts of Features • Drawbacks in Computation Region Growing • Random Walk from discriminative Part • Drawbacks in Compuation Image: [2]: K. Li, Z. Wu, K. -C. Peng, J. Ernst, and Y. Fu. Tell me where to look: Guided attention inference network [3]: K. K. Singh and Y. J. Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization [4]: Y. Wei, J. Feng, X. Liang, M. -M. Cheng, Y. Zhao, and S. Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach Computer Aided Medical Procedures October 17, 2021 Slide 7

Related Work Image Level • Hide Parts of the Image • Drawbacks in Accuracy and Computation Feature Level [5, 6] • Hide Parts of Features • Drawbacks in Computation Region Growing • Random Walk from discriminative Part • Drawbacks in Compuation Image: [5]: X. Zhang, Y. Wei, J. Feng, Y. Yang, and T. Huang. Adversarial complementary learning for weakly supervised object Localization [6]: D. Kim, D. Cho, D. Yoo, and I. S. Kweon. Two-phase learning for weakly supervised object localization Computer Aided Medical Procedures October 17, 2021 Slide 8

Related Work Image Level • Hide Parts of the Image • Drawbacks in Accuracy and Computation Feature Level • Hide Parts of Features • Drawbacks in Computation Region Growing [79] • Random Walk from discriminative Part • Drawbacks in Compuation Image: Simplified from: [9] [7]: R. Fan, Q. Hou, M. -M. Cheng, T. -J. Mu, and S. -M. Hu. s 4 net: Single stage salient-instance segmentation. [8]: A. Kolesnikov and C. H. Lampert. Seed, expand constrain: Three principles for weakly-supervised image segmentation [9]: Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly supervised semantic segmentation network with deep seeded region growing Computer Aided Medical Procedures October 17, 2021 Slide 9

Methodology At a Glance • • • Stochastic Point of View Select hidden Units randomly Explore diverse Combinations of Locations on the Feature Map Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 10

Methodology At a Glance • Expand Feature Map • Center-Fixed Spatial Dropout Selectio • Feature Map -> Classification Score n • Update Network Classifie • Sigmoid Cross Entropy Loss r • For different random Selections • Gradient based CAM Inferenc • Aggregate CAMs e Computer Aided Medical Procedures October 17, 2021 Slide 11

Methodology 1. Stochastic Hidden Unit Selection Goal: Obtain various Classification Scores from a single Image Method: Explore the Classification Score from randomly selected Pairs of hidden Units 1. Feature Map Expansion Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 12

Methodology 1. Stochastic Hidden Unit Selection Goal: Obtain various Classification Scores from a single Image Method: Explore the Classification Score from randomly selected Pairs of hidden Units 1. Feature Map Expansion 2. Center-preserving Dropout [10] Images: Fickle. Net [10]: N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overf Computer Aided Medical Procedures October 17, 2021 Slide 13

Methodology 1. Stochastic Hidden Unit Selection Goal: Obtain various Classification Scores from a single Image Method: Explore the Classification Score from randomly selected Pairs of hidden Units 1. Feature Map Expansion 2. Center-preserving Dropout 3. Generate Receptive Fields of different Shapes and Sizes Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 14

Methodology 1. Stochastic Hidden Unit Selection Goal: Obtain various Classification Scores from a single Image Method: Explore the Classification Score from randomly selected Pairs of hidden Units 1. Feature Map Expansion 2. Center-preserving Dropout 3. Generate Receptive Fields of different Shapes and Sizes 5. Classify with Convolution, then Global Average Pooling and Sigmoid Activation Score S Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 15

Methodology 2. Inference Localization Maps Goal: Single Localization Map Method: Aggregate Maps from random Selections 1. Localization Maps from Grad-CAM [11] 2. Aggregate Localization Maps • Class c if Score for c in any Localization Map at that Pixel u is larger than a Threshold Image: K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition [11]: R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization Computer Aided Medical Procedures October 17, 2021 Slide 16

Methodology • Computer Aided Medical Procedures October 17, 2021 Slide 17

Methodology • [9]: Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly supervised semantic segmentation network with deep seeded region growing Computer Aided Medical Procedures October 17, 2021 Slide 18

Methodology • [9]: Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly supervised semantic segmentation network with deep seeded region growing Computer Aided Medical Procedures October 17, 2021 Slide 19

Methodology At a Glance • • • Stochastic Point of View Select hidden Units randomly Explore diverse Combinations of Locations on the Feature Map Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 20

Methodology Contribution compared to State of the Art Method State of the Art Fickle. Net Selection Image Level Feature Level Random Walk Stochastic Hidden Unit Selection Segmentation Diverse DSRG (State of the Art) Loss Diverse, much handcrafted Handcrafted Computer Aided Medical Procedures October 17, 2021 Slide 21

Experiment Setup Dataset: PASCAL VOC 2012 Image Segmentation Benchmark [12] (20 classes, 10 M labelled images) Network: VGG-16 [13] pretrained on Imagenet [14] Segmentation performed by DSRG [9] Setup: Mini-Batch Size: 10 Image cropped to 321 x 321 Pixels at random Location Learning Rate of 0. 001, halved every 10 Epochs ADAM Optimizer [15] Frame: Pytorch [16] for Localization Maps Caffe [17] for Segmentation [12]: M. Everingham, L. Van Gool, K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge NVIDIA TITAN Xp C. GPU [13]: K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition [14]: J. Deng, W. Dong, R. Socher, L. -J. Li, K. Li, and L. Fei. Imagenet: A large-scale hierarchical image database [9]: Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly supervised semantic segmentation network with deep seeded region growing [15]: D. P. Kingma and J. Ba. Adam: A method for stochastic optimization [16]: A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De. Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch [17]: Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding Computer Aided Medical Procedures October 17, 2021 Slide 22

Experiment Evaluation Baseline Weakly Supervised Setting Semi Supervised Setting • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 • Data: PASCAL VOC 2012 [12] • Segmentation: Res. Net • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 [12]: M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge Computer Aided Medical Procedures October 17, 2021 Slide 23

Results and Discussion Results Weakly Supervised Setting Semi Supervised Setting • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 • Data: PASCAL VOC 2012 [12] • Segmentation: Res. Net • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 [12]: M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge Computer Aided Medical Procedures October 17, 2021 Slide 24

Results and Discussion Results Weakly Supervised Setting Semi Supervised Setting • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 • Data: PASCAL VOC 2012 [12] • Segmentation: Res. Net • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 [12]: M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge Computer Aided Medical Procedures October 17, 2021 Slide 25

Results and Discussion Results Weakly Supervised Setting Semi Supervised Setting • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 • Data: PASCAL VOC 2012 [12] • Segmentation: Res. Net • Data: PASCAL VOC 2012 [12] • Segmentation: Deep. Lab. VGG 16 [12]: M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge Computer Aided Medical Procedures October 17, 2021 Slide 26

Results and Discussion Results Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 27

Conclusion and Future Work Learnings from the Experiment Map Expansion More random Selections of a single Image Dropout Rate • Faster Computing • Little more GPU memory • Larger Areas of the Object are represented • Cover larger Regions of the target Object Computer Aided Medical Procedures General Dropout • Noisy Activation October 17, 2021 Slide 28

Conclusion and Future Work Learnings from the Experiment Map Expansion More random Selections of a single Image Dropout Rate • Faster Computing • Little more GPU memory • Larger Areas of the Object are represented • Cover larger Regions of the target Object General Dropout • Noisy Activation Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 29

Conclusion and Future Work Learnings from the Experiment Map Expansion More random Selections of a single Image Dropout Rate • Faster Computing • Little more GPU memory • Larger Areas of the Object are represented • Cover larger Regions of the target Object General Dropout • Noisy Activation Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 30

Conclusion and Future Work Learnings from the Experiment Map Expansion More random Selections of a single Image Dropout Rate • Faster Computing • Little more GPU memory • Larger Areas of the Object are represented • Cover larger Regions of the target Object General Dropout • Noisy Activation Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 31

Conclusion and Future Work Learnings from the Experiment Map Expansion More random Selections of a single Image Dropout Rate • Faster Computing • Little more GPU memory • Larger Areas of the Object are represented • Cover larger Regions of the target Object General Dropout • Noisy Activation Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 32

Conclusion and Future Work Learnings from the Experiment Image: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 33

Conclusion & Outlook Promising Directions – Using complete stochastic Approach in other Domains – Think about the Loss Functions Computer Aided Medical Procedures October 17, 2021 Slide 34

Conclusion & Outlook Critics + Rethink the entire Process + Possible new Direction for similar Applications - Little outlook for further Research - Some handcrafted Engineering for Improvement Computer Aided Medical Procedures October 17, 2021 Slide 35

Computer Aided Medical Procedures October 17, 2021 Slide 36

Bibliography [1]: B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization [2]: K. Li, Z. Wu, K. -C. Peng, J. Ernst, and Y. Fu. Tell me where to look: Guided attention inference network [3]: K. K. Singh and Y. J. Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization [4]: Y. Wei, J. Feng, X. Liang, M. -M. Cheng, Y. Zhao, and S. Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach [5]: X. Zhang, Y. Wei, J. Feng, Y. Yang, and T. Huang. Adversarial complementary learning for weakly supervised object Localization [6]: D. Kim, D. Cho, D. Yoo, and I. S. Kweon. Two-phase learning for weakly supervised object localization [7]: R. Fan, Q. Hou, M. -M. Cheng, T. -J. Mu, and S. -M. Hu. s 4 net: Single stage salient-instance segmentation. [8]: A. Kolesnikov and C. H. Lampert. Seed, expand constrain: Three principles for weakly-supervised image segmentation [9]: Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly supervised semantic segmentation network with deep seeded region growing [10]: N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting [11]: R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization [12]: M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge [13]: K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition [14]: J. Deng, W. Dong, R. Socher, L. -J. Li, K. Li, and L. Fei. Imagenet: A large-scale hierarchical image database [15]: D. P. Kingma and J. Ba. Adam: A method for stochastic optimization [16]: A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De. Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch [17]: Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding Computer Aided Medical Procedures October 17, 2021 Slide 37

Discussion, Q&A Possible Points to start from: • Significance of the Contribution – Application Perspective – Network Architecture Perspective • Comparability of Results • ? Computer Aided Medical Procedures October 17, 2021 Slide 38

Extra Slide • Computer Aided Medical Procedures October 17, 2021 Slide 39

Extra Slide VGG 16 Network Image: [14]: J. Deng, W. Dong, R. Socher, L. -J. Li, K. Li, and L. Fei. Imagenet: A large-scale hierarchical image database Computer Aided Medical Procedures October 17, 2021 Slide 40

Extra Slide Image-Level Processing High level: Consider the entire image, confuse the classifier by guiding its attention Image-Level hiding and erasure Method: hide random regions of the training image Drawback: semantics and sizes of objects are not considered AE-PSL Method: start with small region in the object, discover new parts and erase old parts Drawback: requires multiple classification networks GAIN Method: CAM trained to confuse the classifier Drawback: Classifier is (too) confused if objects only discriminative parts are erased Computer Aided Medical Procedures October 17, 2021 Slide 41

Extra Slide Feature-Level Processing High level: Classifiers attention based on features ACo. L, TPL Method: Classifier identifies discriminative parts and erases them based on features, 2 nd classifier finds complementary parts of objects from erased features Drawback: requires two networks MDC Method: Ensemble learning with different CAMs, obtained with different dilations Drawback: seperate training procedure for each dilation rate, fixed sized convolution results in false positive regions Computer Aided Medical Procedures October 17, 2021 Slide 42

Extra Slide Region Growing High level: Probability of further regions to also belong to a class Affinity. Net Goal: expand localization map of the CAM Method: random walk based on semantic affinities, which identifies an objects pixels Drawback: additional network for semantic affinities, high dependance on CAM SEC Goal: expand localization map of the CAM constrained Method: new loss function, constrain it to object boundaries Drawback: additional network Computer Aided Medical Procedures October 17, 2021 Slide 43

Extra Slide Grad-CAM Pixe-space gradient Class discriminative Combined Gradvisualization, but CAM with Guided region not class Backprop discriminative Occlusion Visualization for last Sensitivity, similar layer results, expensive to compute Image: Very deep convolutional networks for large-scale image recognition, Simonyan and Zisserman Computer Aided Medical Procedures October 17, 2021 Slide 44

Extra Slide Markov Random Field (MRF) Conditional Random Field (CRF) Source: https: //towardsdatascience. com/conditional-random-fields-explained-e 5 b 8256 da 776 Computer Aided Medical Procedures October 17, 2021 Slide 45

Extra Slide Exact Results Source: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 46

Extra Slide Exact Results Source: Fickle. Net Computer Aided Medical Procedures October 17, 2021 Slide 47