Multispectral Image Analysis for Skin Condition Diagnosis Alison

Multispectral Image Analysis for Skin Condition Diagnosis Alison Wright and Yaning Wang EN. 520. 483. 01. SP 20 Bio-Photonics Laboratory

Clinical Background and Need • Skin is the largest organ in the human body and consists of two principal layers: the epidermis and the dermis. The epidermis is a stratified squamous epithelium, which consists of 4 types of cells: [3] • Melanocytes, Langerhans cells, Merkel cells and Keratinocytes • Malignant melanoma is one of the most rapidly increasing cancers all over the world, with the estimated new cases of 76, 380 and estimated death of 10, 130 in the United States in 2016[1] 2 Fig. 1&2 Anatomy of the skin and images of in sinu melanoma(a), invasive melanoma(b)[3]

Clinical Background and Need • Dermoscopy is a popular in vivo non-invasive imaging tool that uses polarized light to aid dermatologists in examining pigmented skin lesions based on a set of morphological features. • Automatically segmenting melanoma from the surrounding skin is an essential step in computerized analysis of dermoscopic images. Fig. 5 Dermoscopy application 3 Fig. 3&4 Challenges of automated lesion segmentation[1][2]

Previous work: automatic skin segmentation • Unsupervised color image segmentation[4] • Principal Component Analysis (PCA) • Thresholding-based methods rely on the histogram distribution of image color which may be altered by hair and bubble in Ro. I[1]. . • Classic GVF : influenced by distractions or noise • Above methods all employ hand-crafted features that require specialized domain knowledge Table. 1 Comparison of several unsupervised segmentation 4

Previous work: automatic skin segmentation • Supervised image segmentation method using convolutional neural networks • These models have the capability of learning hierarchical features from raw image data without hand-crafted features. • Requires thousand annotated training samples 5 Fig. 7 Image mapping from cell membrane to segmentation map[6] Fig. 6 Structure of unet medical image segmentation[6]

Previous work: Multispectral imaging • The use of skin color in dermatological diagnosis relies mainly on RGB (red, green, blue) color photography. This approach has limits due to the poor color discrimination of the human eye and brain. • Multispectral imaging systems have the capacity to acquire images at different spectral bands, including wavelengths which the human eye is unable to capture. Fig. 8&9 Different multispectral imaging system[5][7] 6

Method • Image preprocessing : hair removal and skin pore removal • Image fusion for multispectral skin images based on wavelet decomposition • Skin Lesion Segmentation using Bi-Directional Conv. LSTM U-Net 7

Image preprocessing • Hair removal – Edge detection & region filling • Modify image preprocessing method[9] • Example from ISIC 2016 dataset 8

Image fusion for multispectral system • Image registration(rigid transform) Image fusion(wavelet based)[8] • Dataset: Oliver Lezoray University of Caen, IUT Grand Ouest Normandie, MMI Dpt. Dermoscopy Skin Lesion Multispectral Image Database • 30 multispectral dermoscopic images (800 x 600) composed of 6 spectral bands (3 in visible light and 3 in infrared -IR- light) Fig. 10&11 Structure of image fusion algorithm[7][8] 9

Image fusion for multispectral system • Image fusion result near infrared reference visible light 10

Network architecture • Bi-Directional Conv. LSTM U-Net[10] Replace 11

Network architecture • • Bi-Directional Conv. LSTM U-Net[10] Bi-Directional Conv. LSTM U-Net Loss function: binary_crossentropy Software: Keras, Tensorflow-GPU backened 30 epoch batch_size 16 Runtime: 40 minutes Input data/Groundtruth: ISIC 2016 Challenge Part 1 500/200 training/validation/testing After image preprocessing 12

Result • Result : input/groundtruth/segmented result 13

Result • Evaluation • compared with participants in ISIC 2016 challenge Participant Accuracy Jaccard Index Sensitivity Specificity Urko Sanchez 0. 950 0. 832 0. 901 0. 960 Mahmudur Rahman 0. 949 0. 884 0. 859 0. 962 Our 0. 932 0. 723 0. 991 14

Reference • [1] Y. Yuan, M. Chao and Y. Lo, "Automatic Skin Lesion Segmentation Using Deep Fully Convolutional Networks With Jaccard Distance, " in IEEE Transactions on Medical Imaging, vol. 36, no. 9, pp. 1876 -1886, Sept. 2017. • [2] A. Wong, J. Scharcanski and P. Fieguth, "Automatic Skin Lesion Segmentation via Iterative Stochastic Region Merging, " in IEEE Transactions on Information Technology in Biomedicine, vol. 15, no. 6, pp. 929 -936, Nov. 2011. • [3] K. Korotkov and R. Garcia, “Computerized analysis of pigmented skin lesions: A review, ” Artif. Intell. Med. , vol. 56, no. 2, pp. 69– 90, 2012 • [4] Hance G, Umbaugh S, Moss R, Stoecker W. Unsupervised color image segmentation: with application to skin tumor borders. IEEE Engineering in Medicine and Biology 1996; 15(1): 104– 11. • [5] Romuald Jolivot, Pierre Vabres, Franck Marzani, Reconstruction of hyperspectral cutaneous data from an artificial neural network-based multispectral imaging system, Computerized Medical Imaging and Graphics, Volume 35, Issue 2, 2011, Pages 85 -88, ISSN 0895 -6111, • [6] Olaf Ronneberger. Email author. Philipp Fischer. Thomas Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015 pp 234 -241 • [7] V. Lukinsone, I. Kuzmina, R. Veilande, E. V. Plorina, D. Bliznuks, K. Bolochko, A. Derjabo, I. Lihacova, J. Spigulis, "Multispectral and autofluorescence RGB imaging for skin cancer diagnostics, " Proc. SPIE 11065, Saratov Fall Meeting 2018: Optical and Nano-Technologies for Biology and Medicine, 110650 A • [8] Dong Han, Zhenhua Guo and D. Zhang, "Multispectral palmprint recognition using wavelet-based image fusion, " 2008 9 th International Conference on Signal Processing, Beijing, 2008, pp. 2074 -2077 • [9] Okuboyejo D, Olugbara O, Odunaike S. Unsupervised restoration of hairoccluded lesion in dermoscopic images. In: Medical Image Understanding and Analysis Conference (MIUA ‘ 2014); London, UK. 2014. pp. 91 -96 • [10] R. Azad, M. Asadi, Mahmood Fathy and Sergio Escalera "Bi-Directional Conv. LSTM U-Net with Densely Connected Convolutions ", ICCV, 2019 15

Thank you! 16