Image Processing and Image Analysis for Medical Visualization

Image Processing and Image Analysis for Medical Visualization (II) Bernhard Preim 1

Classification • Segmentation procedures • • • Introduction Point-oriented Procedures Edge-oriented Procedures Region-oriented Procedures Applications: Segmentation of organs, lesions and vessels Postprocessing with morphologic operators Skeletonization Quantitative analysis of medical image data Implementation of image processing algorithms Bernhard Preim 2

Introduction Bernhard Preim 3

Segmentation: Introduction • Goal: Identification of pixels that meet one homogeneity criterion • Why structures are segmented? • Selective highlighting and fading of structures, high-quality visualization • Basis for a classification (muscles, bones, …) • Basis for further processing (e. g. skeletonization) • Basis for a quantitative analysis (volumetry, form-describing parameters, …) Bernhard Preim 4

Segmentation: Introduction Bernhard Preim 5

Segmentation: Introduction Quality criteria: • Precision (comparison with manual segmentation by human experts) • Speed determines application areas (emergency cases, intraoperative use, …) • Robustness against noise • impacts of disruptions, such as patient movement, metal artifacts, influence of variable pathological changes • Fast feedback • Reproducibility (example: follow-up during MS lesions) Bernhard Preim 6

Segmentation: Introduction • Examples: Identification of pathological changes (tumors), different organs, vessels, bones, catheters, … • Problem: Suitability of segmentation methods strongly depends on: • the applied modalities (CT, MRT, ultrasound, …) • the specific acquisition parameters (contrast agent administration, resolution, noise, sequence, …) • the objects to be detected (thin hierarchical vessels, compact large organs) Bernhard Preim 7

Segmentation: Introduction • Point-oriented Procedures • Edge-oriented Procedures • Region-oriented Procedures • Model-based Procedures Homogeneity Critera: • grey values, gradients, textures Bernhard Preim 8

Segmentation: Point-oriented Procedures • Pixels are attributed to objects based on the gray value (thresholding) • Histogram analysis to identify suitable thresholds (minima) • Advantage: fast, simple method • Disadvantage: Neglect of spatial relations → postprocessing • Variations: local threshold methods (threshold based on a local histogram analysis) Bernhard Preim 9

Segmentation: Point-oriented Procedures • Definition of optimal thresholds by adaptation of probabiliy density distributions to the histogramm • Incorporation of model assumptions • Example: Assumption: three structures are searched for. Intensity values are normally distributed. • Approach: • 1. Estimation of parameters of the normal distribution (mean, standard deviation) based on the histogram. • 2. Estimation of thresholds to minimize the segmentation error. Bernhard Preim 10

Segmentation: Point-oriented Procedures Example: Threshold definition for the segmentation of brain structures in MRI (From: [Hahn, 2001]). Bernhard Preim 11

Segmentation: Point-oriented Procedures Adaptation of 3 Gaussian distributions to the histogram of an MRI brain dataset. Criterion: Minimal quadratic error between histogram and model function (Least Square Fit). Bernhard Preim 12

Segmentation: Point-oriented Procedures • Intensity-based segmentation and definition of regions (connected component analysis, CCA) • Application: segmentation of the kidney from CT data © Pommert [2004] Bernhard Preim 13

Segmentation: Point-oriented Procedures Another example: Bone segmentation by combining thresholding (above) and CCA (below) Bernhard Preim 14

Segmentation: Region-oriented Procedures Frequent Approaches: • Threshold-based region growing based on seed points (Region Growing, Volume Growing) • Watershed transform Bernhard Preim 15

Segmentation: Region Growing • Starting from a seed pixel (white), adjacent pixels with similar intensity are looked for • Problem: Holes in the contour Bernhard Preim 16

Segmentation: Region Growing Region growing with interactive threshold selection. Threshold-volume diagram supports the threshold detection. Strong discontinuities in the diagram are “candidates” for thresholds. (From: Boskamp et al. [2004]) Bernhard Preim 17

Segmentation: Region Growing Problem: Leakage Refinement: Masking of “disturbing” regions (pre-masking) by setting marks. (From: Boskamp et al. [2004]) Bernhard Preim 18

Segmentation: Region Growing Pre-maskings for a subsequent region growing. The target structure (left) is separated from the disturbing structure (right). Source: Boskamp et al. [2004] Bernhard Preim 19

Segmentation: Region Growing Results of vessel segmentation → Video: Hepatic_Vasculature_1, Hepatic_Vasculature_2 Bernhard Preim 20

Segmentation: Region Growing Results of vessel segmentation → Video Vessel. Seg 3 D Movie Bernhard Preim 21

Segmentation: Region Growing • Advanced approaches: • User specifies a direction for the growing process. • Thresholds are adapted to the local intensity distribution. • Consideration of gradient strength. Adding voxels terminates at voxels with high gradient magnitude. • Combination with knowledge representation. Bernhard Preim 22

Segmentation: Region Growing Combination with knowledge representation. Fitting cross sections of an ellipse to avoid leakage. Segmentation of the coronary artery. Source: Hennemuth et al. [2005] Bernhard Preim 23

Segmentation: Region Growing Result of the modified region growing segmentation (From: Hennemuth et al. [2005]) Bernhard Preim 24

Segmentation: Watershed Transform • Idea: Consider the gradient image as mountains that are flooded. Associated catchment basins correspond with one object. • Gray values as elevations; water creates a catchment basin around local minima. Watersheds are created between the catchment basins. • Each catchment basin is assigned a representative value (mean value) –> creation of a mosaic picture. • Smaller (lower) basins are merged via “flooding”. Bernhard Preim 25

Segmentation: Watershed Transform Source: Hahn et al. [2000] Bernhard Preim 26

Segmentation: Watershed Transform Example • Starting point: strong oversegmentation (due to noise in CT data) • stepwise region merging via “flooding” • image as hierarchy of regions (Schindewolf et al. [2000]) Bernhard Preim 27

Segmentation: Watershed Transform Region Merging (Merge Tree) Bernhard Preim 28

Segmentation: Watershed Transform • Exclusion of sub-areas through “exclude” (creation of a new watershed) • Inclusion of sub-areas through “include” (removal of a watershed) Source: Hahn et al. [2004] Bernhard Preim 29

Segmentation: Watershed Transform Segmentation of brain and ventricles Source: Hahn et al. [2000] Bernhard Preim 30

Segmentation: Watershed Transform Source: Hahn et al. (2003) Bernhard Preim 31

Segmentation: Live. Wire • Edge-based procedure is based on a path search (Dijkstra‘s algorithm, http: //www. youtube. com/watch? v=87_1 K 2 GQFd. U) • Searching of paths with minimum costs • Choose the cost function for the path from p to q such that the edge represents the object boundary • Elements of the function: • expected gray values of the target object fz • expected gradient strength fg • edge direction or changes of the direction f. D • Optionally: values of the second order derivative Bernhard Preim 32

Segmentation: Live. Wire • Example for a cost graph Bernhard Preim 33

Segmentation: Live. Wire • Suitability of a cost function is illustrated in the cost image (dark means low costs) Bernhard Preim 34

Segmentation: Live. Wire segmentation of the liver. Will be shown in the exercise! Source: Schenk et al. (2003) Bernhard Preim 35

Segmentation: Live. Wire Summary: Live. Wire is an effective approach • if the target structure is clearly delineated, • if the properties of the target structure and its boundary are rather homogenous, and • If the target structure does not exhibit many identations, such as brain folds. Live. Wire is an intuitive method which is controlled precisely and easily. Organ segmentation in CT is an example where it works well, the segmentation of brain structures in MRI is an example where it is not appropriate. Bernhard Preim 36/62

Segmentation: Model-based Approaches • Idea: Problem-specific know-how is incorporated as constraint • Examples: • • Number of objects (brain: gray and white substance, brain ventricle) Rough size of regions Intensity values and gradient strength Location of objects (in relation to other (presegmented) objects) • Parallel: voice recognition, separation of words • Knowledge about possible words (dictionaries and combinations of words are used) Bernhard Preim 37

Segmentation: Model-based Approaches • Example: atlas-based brain segmentation • Approach: • The brain of a patient is mapped to a segmented (normal) brain • Segmentation of anatomical structures is adopted • Pathological changes are structures that can not be assigned (e. g. lesions in MS patients) Bernhard Preim 38

Segmentation: Model-based Approaches • Segmentation • • Masses are sensors that analyze the image intensity and gradient strength For stability: Constraints for angles, springs with high stiffness Configuration of masses and springs represents the expected form Stable mass-spring models for segmentation of lymph nodes (-> video), vessels, thyroid cartilage and M. sternoclaidomastoideus J. Dornheim et al. , Proc. of MICCAI 2006 and Journal of Academic Radiology 2007 Bernhard Preim 39

Segmentation: Model-based Approaches → Video: Lymph node segmentation → Video: Complete lymph node segmentation → Video: Thyroid cartilage adjustment Bernhard Preim 37

Segmentation: Model-based Approaches Segmentation of blood vessels of the neck J. Dornheim et al. , Proc. of EG Workshop on VCBM Bernhard Preim → Video: Neck. Vessel. Segmentation 41

Segmentation: Interaction • Fully automatic segmentation is rarely performed → manual input before/after application of a segmentation procedure Forms of interaction: • Correction of thresholds • Limitation of the segmentation to one region of interest (ROI) • Line drawings • Stepwise refinement (include, exclude) Postprocessing may be required! Bernhard Preim 42

3 D Object Manipulation • Automatic model-based segmentation often causes partly incorrect results • Incremental and intuitive manipulation of the corresponding 3 D objects • Inspiration: - general 3 D object - manipulation techniques for bulging and bending of 3 D objects (Igarashi‘s Teddy system, Laplacian Modelling) (Bending Tool, Master thesis, D. Proksch [2009], BVM [2010]) Bernhard Preim 43

3 D Object Manipulation Bulging Tool Sketch Tool (Master thesis, D. Proksch [2009], Proksch et al. BVM [2010]) → Videos: Bending. Tool, Bulge. Tool, Sketch. Tool, Traction. Tool Bernhard Preim 44

3 D Object Manipulation Further improvements: • Model information shall be used during the correction → • According to the model the contour is attracted by strong gradients and other characteristics Bernhard Preim 45

Segmentation: Representation of Results Result: • Assignment of voxels to 1 or more objects • Objects may: • overlap • be hierarchically structured (lung tumor) Representation: • 1 bit per object • Segmentation image (1 bit combination per voxel that specifies to which objects the voxel belongs) Bernhard Preim 46

Overlapping Structures • Tumor voxels are also voxels of the liver parenchyma • Infiltrations Bernhard Preim 47

Overlapping Structures • Problem: no explicit voxel assignment any longer • 1 st solution: • Explicit assignment of each structure to 1 bit of an 8 byte value • Limited to 64 structures Bernhard Preim 48

Multicoded Segmentation Masks Refined solution: • Serial creation of a new volume by adding a segmentation mask • Storage of voxel values in a Lookup table Bernhard Preim 49

Multicoded Segmentation Masks Small example. Classification of voxels to 4 objects. Reduction of theoretically 16 possibilities to 7 (reduction of 4 bit to 3). The reduction effect is larger for (realistically) larger examples. (Source: Mühler et al. , 2010) Bernhard Preim 50

Multicoded Segmentation Masks Advantages: • Only 1 volume left • According to the number of overlappings "many" structures are storable in 1 data set • Table: 1 -3 bytes required HLA = neck dissection Bernhard Preim 51

Visualization of Segmentation Results • Standard approach: use of a binary mask (0 background, 1 object) • Visualization via Marching Cubes with threshold value 0. 5 • Problem: Quality, especially in case of large slice distances • Possible solutions: Interpolation of inter layers (e. g. Cubic Interpolation); high storage requirements • Instead of Marching Cubes: Cubic Interpolation Bernhard Preim 52

Visualization of Segmentation Results Arteria carotis. Slice distance: 4 mm. Middle: cubic interpolation of inter layers, right: shape-based interpolation. After interpolation: isotropic data. (Images: Tobias Mönch, Vis. group) Bernhard Preim 53

Visualization of Segmentation Results Visualization of segmentation results as contours. Quelle: Zachow [2004] Bernhard Preim 54

Visualization of Segmentation Results Brain vessels extracted from MRI data. “Normal” and enhanced interpolation. Transfer functions not fully identical. Thus, details are missing (right). Screenshot: Fernando Vega, Universität Erlangen Bernhard Preim 55

Visualization of Segmentation Results • Surface relaxation. Small deformation, which smoothes and minimizes the surface. The binary volume defines constraints (this minimizes the error). Eye with optic nerve. Source: Insight Toolkit Bernhard Preim 56

Visualization of Segmentation Results Left: Marching Cubes Right: Reduction of aliasing artifacts Brain illustration Source: Insight Toolkit Bernhard Preim 57

Morphological Operators: Erosion Processing of ultrasound data with erosion, [Sakas, 1995] Bernhard Preim 62/62

Morphological Operators: Erosion Advanced morphological processing to close holes in a bone Bernhard Preim 63

Skeletonization and Analysis of Image Data • Skeleton: form-describing structure • Discrete skeleton: 1 voxel-wide line that proceeds through the middle of the object. • Motivation: • Basis for measurement (e. g. vessel radii) • Basis for analyzing branchings • Path planning for virtual endoscopy (VL) • Basis for vessel visualization (VL 9) • Skeletonization: • Centers of maximum circular disks • Continued erosion (thinning) Bernhard Preim 64

Skeletoinzation: Illustration • Connection of the centers of maximum circular disks • Grass fire: the outer areas, which remain after burning down the grass fire, form the skeleton Bernhard Preim 65

Skeletoinzation: Approaches • Thinning: gradual erosion of the skeletonized object until a 1 voxel -wide line results. • Search for centers of maximum circular disks • Approach: • Distance transformation calculation for all object points • Determination of the centers of maximum circular disks • Connection of the located centers Bernhard Preim 66

Skeletoinzation: Problems • • Irrelevant side branches (b) Lines that are 2 pixels wide (c) Topology is not preserved after rotation! Many procedures are not (hardly) transferable to 3 d Bernhard Preim 67

Filtering of Irrelevant Side Branches • Criteria: • Absolute length of the side branches • Ratio: length side branch/main branch • Hierarchy position (only lowest level branches can be irrelevant) Bernhard Preim 68

Quantitative Analysis of Medical Image Data Why is a quantitative analysis carried out? • In addition to a visual review • Objektivization of diagnoses and courses of disease • Decision support for continuation/termination of a therapy (chemo, hormone, drug therapy) • Therapy planning (security edges) • Geometrical modeling of implants • Visualization of the quantitative analysis (e. g. bone density) Bernhard Preim 69

Quantitative Analysis of Medical Image Data What is analyzed? • Number, dimension and volume of pathological structures (tumors, infarct areas, herds in MS patients) • Angle between anatomical structures (e. g. to assess malpositions) • Distance between pathological structures and land marks (description of localization) • Distance between pathological structures and organs at risk (for therapy planning) • Gray values of objects (mean value, deviation) Bernhard Preim 70

Quantitative Analysis: Volumetry • Volumetry of compact objects: Segmentation and subsequent multiplication of voxel number and voxel size • Problem: processing of boundary voxels, especially in case of filigree structures with many edge voxels (partial volume effect) • Goal: Classification to objects on a sub-voxel level (Voxel vi: 10 % Obj 1, 90 % Obj 2) Bernhard Preim 71

Quantitative Analysis: Volumetry • Histogram-based analysis of the partial volume effect. If there is a “high” plateau between maxima in the histogram (intensity values that cannot be clearly assigned to a structure), the volumetric uncertainty is huge. • Example: Volumetry of brain ventricles (Schindewolf et al. [1999]) Bernhard Preim 72

Quantitative Analysis: Measurement of the Extent • Goal: Identification of the prolongation of an object o • First idea: Identification of the axis-oriented Bounding Box • Approach: Search through all coordinates and identify minima and maxima in x, y and z direction. • Problem: The calculated prolongation changes as soon as the object is rotated. Bernhard Preim 73

Quantitative Analysis: Measurement of the Extent • Goal: Identification of the maximum diameter of an object o • Idea: Application of a main axis transformation • Approach: Identification of the barycenter and the main axes (mechanical interpretation: axes of inertia) that characterize the orientation of o. • Result: Local, rectangular coordinate system with barycenter as origin and length of the diameter in these new axes (Oriented Bounding Box). Bernhard Preim 74

Quantitative Analysis: Measurement of the Extent 1. Identification of the barycenter vector m 2. Subtraction of the barycenter: pi from P: pĩ = pi – m 3. Identification of covariance matrix A: 1 (pĩx)2 ((pĩx) (pĩy)) A = – ((pĩx) (pĩy)) (pĩy)2 N ((pĩx) (pĩz)) ((pĩy) (pĩz)) ((pĩx) (pĩz) ((pĩy) (pĩz)) (pĩz)2 4. Identification of the Jacobian matrix C, such that A = C-1 AC (these are the normalized eigenvectors of covariance matrix A) 5. Virtual object rotation according to matrix C 6. Calculation of the Bounding Box of the rotated object (OBB) to define the length of the main axes 7. Back-transformation according to C-1 through transposition Bernhard Preim 75

Quantitative Analysis: Measurement of the Extent Calculation principle Example of use (3 d) Bernhard Preim 76

Quantitative Analysis: Distance Transform • Goal: Emanating from a structure (feature points), all pixels shall be assigned the integral distance to this structure. • Result: Distance Map (Source: Lohmann [1998]) • Applications: • Assessment: Which structures are located within a certain edge and another structure? • Question to determine influence areas (e. g. of vessels): Is a voxel closer to structure 1 than to structure 2? • Metrices • Euclidean Metrics: DE [(i, j), (h, k)] = sqrt ((i-h)2 + (j-k)2) • City-Block Distance: D 4 [(i, j), (h, k)] = |i-h| + |j-k| • Chessboard metrics: D 8 [(i, j), (h, k)] = max{|i-h|, |j-k|} Bernhard Preim 77

Quantitative Analysis: Distance Transform Identification of the supply areas of blood vessels Mevis. Lab Euclidean. DTF Bernhard Preim 78

Implementation of Image Processing Algorithms Concepts for efficient image processing: • Division of an image page by page or tile by tile • Processing only after request of a display (Lazy Evaluation) • Caching strategies • Processing of single tiles/pages in separate threads (all such concepts are supported in Me. Vis. Lab; to some extent, the usage requires dedicated programming, e. g. in case of threads) Bernhard Preim 79

Summary • Numerous segmentation methods are necessary to cope with the variety of biological structures and imaging procedures • Generally applicable vs. special procedures • Segmentation often as core of a pipeline with preprocessing of the images and postprocessing of the segmentation results • Skeletonization is an image analysis step that is based on segmentation • Often, the quantitative analysis is a decisive advantage of digital image analysis Application: Specification of diagnoses and courses of disease. Bernhard Preim 80

Literature W. Abmayr. Einführung in die digitale Bildverarbeitung, Teubner-Verlag Stuttgart 1992 T. Boskamp, D. Rink, F. Link et al. New Vessel Analysis Tool for Morphometric Quantification and Visualization of Vessels in CT and MR Imaging Data Sets, Radiographics, 2004, Heft 2 J. Dornheim, D. Lehmann, L. Dornheim, B. Preim and G. Strauß. „Reconstruction of Blood Vessels from Neck CT Datasets using Stable 3 D Mass-Spring Models“, Proc. of EG Workshop on Visual Computing in Biology and Medicine, S. 77 -82, 2008 J. Dornheim, H. Seim, B. Preim, I. Hertel and G. Strauss. „Segmentation of Neck Lymph Nodes in CT Datasets with Stable 3 D Mass-Spring Models“, Proc. of Medical Image Computing and Computer-Assisted Intervention (MICCAI), S. 904 -911, 2006 J. Dornheim, H. Seim, B. Preim, I. Hertel and G. Strauß. „Segmentation of Neck Lymph Nodes in CT Datasets with Stable 3 D Mass-Spring Models“, Special Issue of Academic Radiology, S. 1389 -1399, 2007 H. K. Hahn, H. -O. Peitgen. “The Skull Stripping Problem In MRI Solved By A Single 3 D Watershed Transform”, Medical Image Computing and Computer-Assi-sted Intervention (MICCAI'2000), Pittsburgh, Springer, 2000, S. 134 -143 H. K. Hahn and H. -O. Peitgen (2003). , “IWT – Interactive Watershed Transform: A hierarchical method for efficient interactive and automated segmentation of multidimensional grayscale images”, Proc. SPIE Medical Imaging, SPIE Band 5032, Bernhard Preim 81

Literature A. Hennemuth, S. Bock, T. Boskamp, D. Fritz, C. Kuehnel, D. Rinck, M. Scheuering, “One-click coronary tree segmentation in CT angiographic images, ” in Proc. of Comput. Assist. Radiol. Surg. (CARS’ 05), 2005, S. 317– 321 A. Hennemuth. Computer-Assisted Diagnosis and Therapy Planning Coronary Artery Disease Based on Cardiac CT and MRI. Ph. D thesis, Department for Mathematics and Computer Science, University Bremen, 2012 H. Lamecker, T. Lange, and M. Seebass. A Statistical Shape Model for the Liver. Proc. of Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 2488 of Lecture Notes in Computer Science, pages 422– 427, Springer, 2002 Th Lehmann, W Oberschelp, E Pelikan und R Repges. Bildverarbeitung für die Medizin, Springer-Verlag, Heidelberg, 1997 G. Lohmann. Volumetric Image Analysis, Wiley und Teubner, 1998. K. Mühler, C. Tietjen, F. Ritter and B. Preim (2010). „The Medical Exploration Toolkit: An Efficient Support for Visual Computing in Surgical Planning and Training, IEEE Transactions on Visualization and Computer Graphics, S. 133 -146 D. Proksch, J. Dornheim, B. Preim. „Interaktionstechniken zur Korrektur medizinischer 3 DSegmentierungen“, Bildverarbeitung für die Medizin (BVM) (420 -424), 2010 Bernhard Preim 82

Literature G. Sakas and S. Walter. Extracting surfaces from fuzzy 3 d-ultrasound data. In Proc. of ACM SIGGRAPH, pages 465– 474, 1995 A. Schenk, G. Prause, H. O. Peitgen (2000). Efficient Semiautomatic Segmentation of 3 D Objects in Medical Images. Proc. of MICCAI 2000: 186 -195 D. Selle. Analyse von Gefäßstrukturen in medizinischen Schichtdatensätzen für die computergestützte Operationsplanung, Dissertation, Universität Bremen, 1999 T. Schindewolf, U. Freese, J. Meißner. “Segmentierung und Volumetrie der Hinrventrikel mit MRT-Datensätzen”, Bildverarbeitung für die Medizin, Springer-Verlag, 1999, S. 92 -96 T. Schindewolf, H. -O. Peitgen. “Interaktive Bildsegmentierung von CT- und MR-Daten auf Basis einer modifizierten Wasserscheidentransformation”, Bildverarbeitung für die Medizin, 2000 J. G. Sled, A. P. Zijdenbos, and A. C. Evans. A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Transactions on Medical Imaging, 17(1): 87– 97, 1998 F. Vega, N. Sauber, B. Tomandl, C. Nimsky, G. Greiner, and P. Hastreiter. Automatic Adjustment of Bidimensional Transfer Functions for Direct Volume Visualization of Intracranial Aneurysms. In Proc. of SPIE Medical Imaging, volume 5367 of LNCS, pages 275– 284, 2004 S. Zachow, H. -C. Hege, and P. Deuflhard. Computer assisted planning in cranio-maxillofacial surgery. Journal of Computing and Information Technology, 14(1): 53– 64, 2004 Bernhard Preim 83