Chair for Computer Aided Medical Procedures Augmented Reality

Chair for Computer Aided Medical Procedures & Augmented Reality Master Seminar: Deep Learning for Medical Applications Multi-task learning for the segmentation of organs at risk with label dependence by Tao He, Junjie Hu, Ying Song, Jixiang Guo, Zhang Yi Martin Axelrad Tinoco Tutor: Magda Paschali

Chair for Computer Aided Medical Procedures & Augmented Reality Master Seminar: Deep Learning for Medical Applications Introduction

Radiation therapy § Damage DNA of cancerous cells § High energy rays or radioactive materials § 60% of cancer patients will require RT § Precise delineation is critical § Manual delineation is expensive, time consuming and lacks of reproducibility Unknown Photographer. (2001). Proton Beam Therapy [A proton beam from the brain during CT scans. Using dozens of CT slices, a computer produced this three-dimensional representation of the eyes and optic nerves (blue and green). The brain stem (green) , and the tumor (red). The yellow line shows the proton beam field-shaping aperture. ]. Retrieved 2020, from https: //visualsonline. cancer. gov/details. cfm? imageid=2421 Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 3

Related work § Multi-task learning in deep neural networks (MTL) Ø Parallel learning - Improves generalization Ø Parameter sharing methods (Hard, soft) Ø Dependent multi-label classification § Two-step methods Ø Cascade learning strategy - concatenated CNN Ø Fast and coarse followed by slow but accurate results Ø Reduce computing time – increased training time Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 4

Related work: Ensemble voting methods Generalizes better than a single model § Models with significant diversity (acquisition protocols) § Training of multiple FCNs followed by voting § Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 5

Goal § What? Ø Performance comparable to state of the art methods Ø Reduce time complexity of previous approaches Ø Without increasing network capacity § How? Ø MTL architecture + combined loss function (EDN learns classification and segmentation) Ø False positive filtering (FPF) algorithm + dynamic threshold selection (DTS) Ø TAOWCH dataset. Real medical organ dataset. Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 6

Chair for Computer Aided Medical Procedures & Augmented Reality Master Seminar: Deep Learning for Medical Applications Methodology

Multi-task learning with encoder-decoder network Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 8

Encoder § Flexibility deploying ensemble with Res. Net or Dense. Net blocks § Last transition block of Dense. Net omitted Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 9

Decoder § Asymmetric architecture § Segmentation performance not degraded § Reduced number of parameters Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 10

False positive filtering with dynamic threshold selection § TPR of classification >> DICE of segmentation § If organ present and segmented, don’t modify the mask 0. 82 0. 54 > 0. 5 0. 67 0. 13 MTL Segmentation architecture Master Seminar: Deep Learning for Medical Applications FPF July 9, 2020 Slide 11

False positive filtering with dynamic threshold selection § TPR of classification >> DICE of segmentation § If organ absent and segmented (FP), discard the mask 0. 82 0. 54 > 0. 5 0. 32 0. 13 MTL Segmentation architecture Master Seminar: Deep Learning for Medical Applications FPF July 9, 2020 Slide 12

False positive filtering with dynamic threshold selection § Problems with FPF Ø Filtering TPs if TPR(classification) >> DICE(segmentation) Ø Checkpoints saved only with segmentation improvement § Dynamic Threshold Selection (DTS) Ø Threshold changes every epoch using ROC curve DTS threshold 1 threshold 2 threshold 3 tmin 0. 82 0. 63 σi 0. 51 0. 13 https: //commons. wikimedia. org/wi ki/File: Roccurves. png Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 13

Weighted mean cross entropy loss function § Organ dependency in CT images → Boost classification performance • Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 14

Segmentation loss: Soft Dice (SD) loss • Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 15

Chair for Computer Aided Medical Procedures & Augmented Reality Master Seminar: Deep Learning for Medical Applications Experiments

Datasets: seg. THOR § § § 40 labeled thoracic CT scans (esophagus, heart, trachea, aorta) CT slices: 512 x 512 pixels. Spatial resolution: 0. 9 - 2. 7 mm Z-axis: 150 - 284 images. Thickness: 2 - 3. 7 mm Cross validation: 4 groups (10 scans ~ 1822 images) Intensity range: - 384 to 384 HU Lambert, Z. , Petitjeanb, C. , Dubrayb, B. , & Ruanb, S. (2019). Seg. THOR: Segmentation of Thoracic Organs at Risk in CT images. LMI, INSA Rouen, France. Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 17

Datasets: TAOWCH 49 labeled thoracic and abdominal CT scans (heart, liver, lung, kidney) § Main challenge: Images captures after cancer surgery § Split of dataset Ø Training: 31 scans (4299 images) Ø Validation: 8 scans (1006 images) Ø Testing: 10 scans (1231 images) § Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 18

Experiment setup and implementation § § § § § Image configuration: 2. 5 D CT images (3 adjacent axial slices) Optimization method: SGD + momentum 0. 9 Learning rate: 0. 01 + decay 0. 95 / epoch Loss function segmentation: Soft dice loss function Loss function classification Ø Binary cross entropy (BCE) Ø Weighted mean cross entropy (WMCE) FPF + DTS: tmin = 0. 995 Data augmentation Ø Horizontal/vertical random flipping Ø Cropping factor: 0. 6 - 1 Metrics: Ø Classification: macro. TPR (average of TPR of all classes) Ø Segmentation: Jaccard, Dice, Hausdorff and FPR Training time: 8 h (seg. THOR) / 7 h (TAOWCH) on 4 x Nvidia Titan XP GPUs Validation, testing: Ensemble voting for final segmentation Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 19

Experimental analysis about transfer learning § § Transfer learning (TL) strategy with pre-trained networks on Image. Net Two EDN: Ø Dense. Net 212 encoder Ø Res. Net 101 encoder With or without TL MTL with TL accelerates the training and boosts the performance Training loss curves with or without transfer learning. The abscissa corresponds to the training steps with respect to mini-batch. Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 20

Experimental results of multi-label classification TPR on encoder-decoder network with Res. Net 101 and Dense. Net 121 encoders (seg. THOR). Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 21

Performance improvements of WMCE (seg. THOR) Experimental results on Seg. THOR dataset. Different configurations using multi-task learning (MTL), false positive filtering (FPF), dynamic threshold selection (TDS), weighted mean cross entropy (WMCE) loss. Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 22

Performance improvements of WMCE (seg. THOR) Segmentation results on seg. THOR Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 23

Performance improvements of WMCE (TAOWCH) Experimental results on TAWOCH dataset. Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 24

Performance improvements of WMCE (TAOWCH) Segmentation results on TAWOCH Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 25

Case studies of label dependence Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 26

Multi-class with WMCE vs single-class § § Single class + ensemble voting: first place at ISBI 2019 Seg. THOR challenge MTL structure with FPF + DTS and WMCE with Res. Net and Dense. Net encoders Han, M. , Yao, G. , Zhang, W. , Mu, G. , Zhan, Y. , Zhou, X. , Gao, Y. , 2019. Segmentation of CT thoracic organs by multi-resolution vb-nets. In: Proceedings of the Challenge on Segmentation of THoracic Organs at Risk in CT Images. Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 27

Chair for Computer Aided Medical Procedures & Augmented Reality Master Seminar: Deep Learning for Medical Applications Discussion and conclusion

Conclusion of the paper § § The combined result (MTL + ED + FPF + DTS + WMCE) improves performance overall Segmentation time Ø Comparable to single model segmentation Ø Outperforms widely used two-step and ensemble voting methods WMCE improves performance of multi-label classification Ø Heavily dependent classes Ø Similar global conditional probability between training and testing set Future work Ø Joint distribution for conditionally dependent variables among all organs Ø Apply WMCE to other MLC tasks (text, image, acoustic) Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 29

Personal conclusions § Pros Ø Rather intuitive but effective proposal Ø Experimental environment and parameters fully specified Ø Thorough validation process § Cons Ø They don’t explain their preliminary study in depth. § Use of FPF as the TPR of classification is greater than DICE for segmentation § Shallow decoder doesn’t affect segmentation performance Ø Code and TAOWCH dataset not publicly available Ø Conclusions mostly is a summary of the previous steps Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 30

Thank you!

Related work: Two-step method § § § Cascade learning strategy Reduce overall computing time Two concatenated FCNs: Ø Step 1: Quick and coarse segmentation of organ Ø Step 2: Accurate contour of the lesion Li, X. , Chen, H. , Qi, X. , Dou, Q. , Fu, C. , & Heng, P. (2018). H-Dense. UNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes. IEEE Transactions on Medical Imaging, 37(12), 2663 -2674. doi: 10. 1109/tmi. 2018. 2845918 Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 32

Problem Statement § Manual delineation has disadvantages Ø Tedious and time consuming for the specialist Ø Human-computer interaction increases workload Ø Subjective based on the experience of radiologist Ø Lacks of reproducibility § What about automatic segmentation? Ø FCNs have become the standard in recent years Ø Require large of labeled datasets Ø Time consuming § Training very large architectures § Volumetric image to segment § Every voxel must be labeled Ø Sensible to image quality - acquisition protocols Master Seminar: Deep Learning for Medical Applications July 9, 2020 Slide 33