Spatially Constrained Segmentation of Dermoscopy Images Howard Zhou

Spatially Constrained Segmentation of Dermoscopy Images Howard Zhou 1, Mei Chen 2, Le Zou 2, Richard Gass 2, Laura Ferris 3, Laura Drogowski 3, James M. Rehg 1 1 School of Interactive Computing, Georgia Tech 2 Intel Research Pittsburgh 3 Department of Dermatology, University of Pittsburgh 1

Skin cancer and melanoma l Skin cancer : most common of all cancers 2 [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Skin cancer and melanoma l l Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%) 3 [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Skin cancer and melanoma l l l Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%) Early detection significantly reduces mortality 4 [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy Clinical View view 5 [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy l l l Improve diagnostic accuracy by 30% in the hands of trained physicians May require as much as 5 year experience to have the necessary training Motivation for Computer-aided diagnosis (CAD) in this area Clinical view Dermoscopy view 6

First step of analysis: Segmentation l l Separating lesions from surrounding skin Resulting border l l l Gives lesion size and border irregularity Crucial to the extraction of dermoscopic features for diagnosis Previous Work : l l PDE approach – Erkol et al. 2005, … Histogram thresholding – Hintz-Madsen et al. 2001, … Clustering – Schmid 1999, Melli et al. 2006… Statistical region merging – Celebi et al. 2007, … 7

Domain specific constraints l Spatial constraints l l Four corners are skin (Melli et al. 2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image Cartesian coordinates (Meanshift) 8

Domain specific constraints l Spatial constraints l l Four corners are skin (Melli et al. 2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image Cartesian coordinates (Meanshift) Meanshift (c = 32, s = 8) 9

We explore … l Spatial constraints arise from the growth pattern of pigmented skin lesions Meanshift (c = 32, s = 8) 10

We explore … l Spatial constraints arise from the growth pattern of pigmented skin lesions – radiating pattern Meanshift (c = 32, s = 8) 11

Embedding constraints l l Radiating pattern from lesion growth Embedding constraints as polar coords improves segmentation performance Meanshift (c = 32, s = 8) Polar (k = 6) 12

Embedding constraints l l Radiating pattern from lesion growth Embedding constraints as polar coords improves segmentation performance Meanshift Polar (k = 6) 13

Comparison to the Doctors l l Radiating pattern from lesion growth Embedding constraints as polar coords improves segmentation performance White: Dr. Ferris Red : Dr. Zhang Blue : computer Meanshift Polar 14

Dermoscopy images Common radiating appearance 15

Growth pattern of pigmented skin lesions l lesions grow in both radial and vertical direction Skin absorbs and scatters light. Appearance of pigmented cells varies with depth l l Dark brown tan blue-gray Common radiating appearance pattern on skin surface [ Image courtesy of “Dermoscopy : An Atlas of Surface Microscopy of Pigmented Skin Lesions] 16

Radiating growth pattern on skin surface l Difference in appearance: more significant along the radial direction than any other direction. 17

Radiating growth pattern on skin surface l Difference in appearance: more significant along the radial direction than any other direction. 18

Embedding spatial constraints Feature vectors l Each pixel feature vector in R 4 l l 3 D: R, G, B or L, a, b in the color space 1 D: polar radius measured from the center of the image (normalized by w) original r {R, G, B} 19

Embedding spatial constraints Grouping features l l l Each pixel feature vector in R 4 Clustering pixels in the feature space Replace pixels with mean for compact representation original filtered r {R, G, B} 20

Radiating pattern Dermoscopy vs. natural images … Derm dataset (216) … BSD dataset (300) 21

Embedding spatial constraints Grouping features Cartesian l Mean per-pixel residue: average per-pixel color difference of each pair {Rc, Gc, Bc} original {Ro, Go, Bo} polar {Rp, Gp, Bp} 22

Dermoscopy vs. natural images Polar vs. Cartesion l Mean per-pixel residue (k-means++, k = 30) Residue (Cartesian) Residue (polar) Derm dataset (216) Residue (Cartesian) Residue (polar) BSD dataset (300) 23

Dermoscopy vs. natural images Polar vs. Cartesion l Mean per-pixel residue (k-means++, k = 30) 24

Polar vs. Cartesian l The regions appear more blocky in the Cartesian case Polar (k = 30) Cartesian (k = 30) 25

Six super-regions l 30 clusters 6 super clusters (K-means++) Polar (k = 6) Cartesian (k = 6) 26

Final segmentation Polar Cartesian 27

Polar vs. Meanshift l The regions appear more blocky in the Meanshift case Polar (k = 6) Meanshift (c = 32, s = 8) 28

Final segmentation Polar Meanshift 29

Algorithm overview l Given a dermoscopy image 30

Algorithm overview l Given a dermoscopy image original 31

Algorithm overview 1. First round clustering: K-means++ (k = 30) original 30 clusters 32

Algorithm overview 2. Second round: clusters(30) super-regions(6) original 30 clusters 6 Super-regions 33

Algorithm overview 3. Apply texture gradient filter (Martin, et al. 2004) original 30 clusters Texture edge map 6 Super-regions 34

Algorithm overview 4. Find optimal boundary (color+texture) original 30 clusters 6 Super-regions Texture edge map 35 Final segmentation

1. First round clustering l First round clustering: K-means++ (k = 30) l l Reduce noise Groups pixels into homogenous regions – a more compact representation of the image Artuhur and Vassilvitskii, 2007 R 4 : {L*a*b* (3 D), w * polar radius (1 D)} original 36

1. First round clustering l First round clustering: K-means++ (k = 30) l l Reduce noise Groups pixels into homogenous regions – a more compact representation of the image Artuhur and Vassilvitskii, 2007 R 4 : {L*a*b* (3 D), w * polar radius (1 D)} original 30 clusters 37

2. Second round clustering l K = 6 : clusters(30) super-regions(6) l l l Account for intra-skin and intra-lesion variations Avoid a large k Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc. original 30 clusters 38

2. Second round clustering l K = 6 : clusters(30) super-regions(6) l l l Account for intra-skin and intra-lesion variations Avoid a large k Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc. original 30 clusters 6 super-regions 39

3. Color-texture integration l Incorporating texture information can improve segmentation performance. l Severely sun damaged skin; texture variations at boundaries in addition to color variations original 40

3. Color-texture integration l Incorporating texture information can improve segmentation performance. l l Severely sun damaged skin; texture variations at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004) original 41

3. Color-texture integration l Incorporating texture information can improve segmentation performance. l l l Severely sun damaged skin; texture variations at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004) Texture edge map: pseudo-likelihood original Texture edge map 42

4. Optimal boundary l Optimal skin-lesion boundary l Color: Earth Mover’s Distance (EMD) between every pair of super-regions 6 super-regions 43

4. Optimal boundary l Optimal skin-lesion boundary l l Color: Earth Mover’s Distance (EMD) between every pair of super-regions Texture: Texture edge map 6 super-regions Texture edge map 44

4. Optimal boundary l Optimal skin-lesion boundary l l l Color: Earth Mover’s Distance (EMD) between every pair of super-regions Texture: Texture edge map Minimizing the integrated color-texture measure 6 super-regions Texture edge map 45

Validation and results l l Our collaborating dermatologist Dr. Ferris manually outline the lesions in 67 dermoscopy images The border error is given by Computer : binary image obtained by filling the automatic detected border ground-truth : obtained by filling in the boundaries outlined by Dr. Ferris 46

Typical segmentation result Error = 12. 96% White: Dr. Ferris Red : Dr. Zhang Blue : computer 47

Comparison To account for inter-operator variation, we also asked Dr. Alex Zhang to manually outline boundaries on the same dataset 48

Additional results White: Dr. Ferris Red : Dr. Zhang Blue : computer Error = 5. 80% 49

Additional results White: Dr. Ferris Red : Dr. Zhang Blue : computer Error = 13. 61% 50

Additional results White: Dr. Ferris Red : Dr. Zhang Blue : computer Error = 16. 60% 51

Additional results White: Dr. Ferris Red : Dr. Zhang Blue : computer Error = 34. 09% 52

Limitation l l Assumption that lesions appear relatively near the center may not hold Fairly low number of super regions (6) may limit the algorithm to perform well on lesions with more colors 53

Conclusion l l l Growth pattern of pigmented skin lesions can be used to improve lesion segmentation accuracy in dermoscopy images. An unsupervised segmentation algorithm incorporating these spatial constraints We demonstrate its efficacy by comparing the segmentation results to ground-truth segmentations determined by an expert. 54

Future work l Extend to meanshift? 55

Comparison to other methods 56

Color and texture cue integration l l Apply texture gradient filter (Martin, et al. 2004) Pseudo-likelihood map - edge caused by texture variation is present at a certain location 57