A Survey of StateofArts Retina Image Registration Methods

A Survey of State-of-Arts Retina Image Registration Methods Presented By: Jian Shi University of South Carolina

Overview n n Motivation Methods Conclusion Future work 2022/2/15 2

Motivation n Motivation q q 2022/2/15 Retina image processing is greatly required in diagnosing and treatment of many diseases affecting retina. The registration of retina images is very useful in helping physicians to do a reliable diagnosis as composing a complete retina map Many research paper talk about various retina registration schemes using different techniques and algorithms. It is still under rapid development, new requirements and new methods continue My intention is to do a survey research of these methods, trying to categorize them, making comparison between them, then see what’d be useful and valuable for future study and research 3

Motivation n The basic several steps of image registration n n Feature detection Feature matching Transform modal estimation Image resampling and transformation Feature detection and matching n n 2022/2/15 The key components of retina image registration Detecting landmarks based on blood vessels and cross points consists the majority group of methods New schemes that are going beyond this boundary I am trying to introduce them in a developing point of view 4

Segmentation before Registration n Popular methods q Detection of blood Vessel boundaries by the difference operators n n q Boundary detection using image statistics n 2022/2/15 Sobel operators, smoothing effect. See “Rafael C. Gonzalez, Richard E. Woods, “Digital image processing”, second edition, Prentice Hall, pp 567 -585, 2002” Caney edge detector, good at detecting blood vessels’ boundaries. See Bill Green, “Canny Edge Detection Tutorial”, http: //www. pages. drexel. edu/~weg 22/can_tut. html, 2002. See E. Aniram, H. Aydinoglu, and I. C. Goknar, “Decision Based Directional Edge Detector, ” Signal Proc. , vol. 35, pp. 149 -156, 1993. 5

Segmentation before Registration n More q Extraction of blood vessel boundaries using deformable models (snake model) n q q See T. Mc. Inerney and D. Terzopoulos, “Deformable Models in Medical Image Analysis: a survey, ” Med Image Analysis, vo. 1, no. 2, pp. 91 -108, 1996. Boundary Detection Using Morphological Gradient Extraction of the Core Area of the Blood Vessel Tree by Matching the Image with Gaussian Filter Segmentation Using Watersheds Extraction of Blood Vessel Tree Using the Morphological Reconstruction Etc… 2022/2/15 6

Vessels are reliable landmarks in retinal images because they are almost rigid structures and they are depicted in all modalities. In the following part I will introduce 2 representative registration method using vessel structures as their features. I don’t talk about the detail of feature extraction because I assume it is well resolved 2022/2/15 7

A Multimodal Registration n Main idea n n Assumptions: q input retina images are restricted to central images of the retina containing the macula, the papilla, and temporal vessels in order to limit deformations. Detecting vessel structures and extracting bifurcation points as features q q n A global affine transform then performed q 2022/2/15 Vessels has to be connected Have a fixed Width less than a threshold and are locally linear Given M is a bifurcation point that to be transformed into M’ using 8

A Multimodal Registration n Invariant properties q q q 2022/2/15 Angles between edges of bifurcation points are preserved The point M can be defined as M=(m, a 1, a 2, a 3, a 4), a 1~4 are the possible 4 directions of a bifurcation point A matching example a counter example 9

A Multimodal Registration n Matching algorithm n n n Drawbacks n n A Bayesian Hough transform is used q That is to decompose affine transformation into a translation S, a rotation R, a homothetie T Each pair of feature points could generate a set of transformation matrix, calculate among 20 sets of matrix to satisfy a min-square estimation Too many constraints and assumptions on inputs Heuristic threshold settings Can’t deal with many types of retina images Advantages n n 2022/2/15 It well defined the invariant features using bifurcation points It has refinement process 10

Using Creases as Landmark n Main idea n n Treat vessels as creases (ridges or valleys) when images are seen as landscapes. Extract the invariant property: we define creases using level set extrinsic curvature (LSEC, see A. L´opez, D. Lloret, J. Serrat, and J. Villanueva. Multilocal creasness based on the level set extrinsic curvature. Computer Vision and Image Understanding, (77), 2000. ). n The level set of a constant level curvature L consists of a set of points X that L(X) = L Matching scheme: q 2022/2/15 Start from an initial guess, transform creaseness image g to f until it’s properly aligned, use the correlation function to check the quality of alignment 11

Using Creases as Landmark When Ct reach a maximum value, it is the best alignment q Here f and g are not the whole image, they’re “creaseness image” that contains pixels with creaseness values higher than a threshold Advantages: q q n n Fast processing speed, since the pixels involved in computation are only a part of the whole image High accuracy, since the quality function is iteratively checked and could reach a global maximum Drawbacks: q n n Need a good initial guess Can’t deal with images without clear vessel structure Retinal image registration using creases as anatomical landmarks David Lloret, Joan Serrat, Antonio M. L´opez, Andr´es Soler, Juan J. Villanueva 2022/2/15 12

The previous two methods are representative because they can be categorized into two different groups: The former one uses geometric transformation to do the registration. And the latter one uses similarity detection by correlation to register retina images. However, their performance are based on the quality of input retina images, yet in reality, there exist many unclear or ill-formed retina images. In those cases, the pervious two methods are not enough. Next I’ll introduce a novel scheme called dualbootstrap published In 2002, for better understanding, we talk about ICP first on which dual-bootstrap is based 2022/2/15 13

Point-based Registration n ICP (Iterative Closest Point) q Explanation n q “Point” here means raw measurements that locally summarize the geometric structure of the data. Such as (x, y, z) values from range images, intensity points in three-dimensional medical images, and edge elements, corners and interest points. Main steps n Given two dataset i, j and T (the parameter vector of the transformation mapping the coordinate system of i onto the coordinate system of j). Repeat: q q n 2022/2/15 using a fixed estimate, T, the transformation is applied to each point from image dataset i and the closest point in image dataset j is found as a temporary match using constraints formed from these matches, a new best T’ is computed Until T stabilizes, that is, algorithm converges 14

Point-based Registration n ICP’s Problems n converge to an incorrect final registration starting from an initial estimate that locally appears correct. q q 2022/2/15 Same image viewed from different perspective (a), (b) are used in initializing ICP; (c), (d) are used in constraining ICP in iterations. 15

Point-based Registration n An example of ICP’s failure q q 2022/2/15 (a) shows the initial alignment based on the single correct correspondence (b) shows the final result after convergence 16

Dual Bootstrap ICP n Dual bootstrap ICP q Three enhancement to original ICP n n n q Main steps n n 2022/2/15 The bootstrap region, it’s a small region at the beginning and gradually increase to the entire image Robust ICP, with a carefully estimated error scale Bootstrapping the model, different models for different regions Start with a initial bootstrap region Repeat applying robust ICP only in the bootstrap region, when the robust ICP converges, increase the bootstrap region size and do it again Until the bootstrap region covers the whole image The transformation model is automatically selected during the size of bootstrap region increasing 17

n Robust ICP q Objective function to be minimized Dual Bootstrap ICP n Ө is the transformation parameter vector; M is the mapping from p 1 in I 1 into I 2, q is the corresponding point in I 2, D is the distance metric between p and q, ρis a loss function, σis the error scale factor, they both are for help rejecting mismatches Mathematical details about the equation above could be found in the original paper “The Dual-Bootstrap Iterative Closest Point Algorithm with Application to Retinal Image Registration Charles V. Stewart Chia-Ling Tsai Badrinath Roysam” 2022/2/15 18

Dual Bootstrap ICP n Automatic model selection 2022/2/15 19

Dual Bootstrap ICP n Bootstrap region increment q A configurable parameter b n n n Move out the perpendicular of each side by b, the default value is sqrt(2) – 1. to make sure in each iteration, the region doesn’t expand to its double size Different value of b will affect the performance of the whole algorithm Invariant properties n 2022/2/15 Angles between edges of bifurcation points, very similar to the first introduced method 20

Dual Bootstrap ICP 2022/2/15 21

Dual Bootstrap ICP n Advantages n n Very high accuracy, 97% success rate both for healthy and pathologic retina. Data gathered from over 6, 000 images Robust to retina image noise and pathologic retina images, which might not have very clear vessel structures could be processed well Average 5 second per image pair Drawbacks n n n 2022/2/15 Initial regions still need human interaction Processing speed various largely by the area of initial region’s size Possible failures still exist 22

Dual-bootstrap is a proven very accurate and sophisticated scheme, but it still limited by image quality, although greatly improved comparing to previous schemes. SURF is a feature detection and description method that proposed in 2006, it totally puts aside the vessel structures, crossing points. So the quality of input image is not going be affect the registration results 2022/2/15 23

SURF n Surf detectors n Based on a Hessian matrix n σis a value that changes by different filter size, 1. 2 when in a 9 x 9 filter, 3. 6 when in a 27 x 27 filter. The determinant of Hessian matrix of the interesting point would remain same while filter size increasing. Scale invariance: use a 3 x 3 neighbor space interpolation, and then the maxima of the Hessian matrix are interpolated in scale using the method in “Brown, M. , Lowe, D. : Invariant features from interest point groups. In: BMVC. (2002)” n 2022/2/15 24

SURF q Rotation invariance: n n n 2022/2/15 firstly, calculate the Haar wavelet response in x, y direction around a circular neighbor area around the interesting point. The dominant orientation is estimated by calculating the sum of all responses within a sliding orientation window covering an angle of π/3. The horizontal and vertical responses within the window are summed. The two summed responses then yield a new vector. The longest such vector lends its orientation to the interest point. The window’s size is a experimental parameter 25

SURF n Advantages q q q n Surf’s blob-like invariant features can totally overcome the mismatch caused by non-clear vessel structures, it totally doesn’t care about vessels, bifurcation points, etc. Surf uses integral image techniques and achieves very good performance It needs no human interaction, so it can be applied in automatic retina image registration Drawbacks q It still fails if the image pairs have poor overlap n 2022/2/15 in experiment, it fails for image pairs that have lower than 14% overlapping part 26

For better efficiency and higher accuracy, automatic retina image registration is in the need. Schemes using SURF method can achieve this goal. Before it, there’re other automatic retina image registration schemes 2022/2/15 27

Auto by Global Optimization Method n Preprocessing q q n Vessel structures are extracted Input image type can only be: Red-Free (RF), Fluoroscein Angiography (FA) and Indocyanine Green Chorioangiography (ICG). That is, they all are easy to be transferred into binary format Choosing the best transformation q T are 3 kinds of transformations: affine, bi-linear and projective transformation 2022/2/15 28

Auto by Global Optimization Method n Determining the transformation parameters q Use three different optimization methods to calculate different weight values for different methods, respectively 2022/2/15 29

Auto by Global Optimization Method n Drawbacks q q q n Need preprocessing Low effect using global optimization search Need input to be well formed Advantages q q It is in 1999 The result is more accurate than manual registration 2022/2/15 30

q q q An referencing retina image spatial map is pre-calculated Transformation model is fixed similarity transformation Constellation: landmark pairs (constraints: pairs used as constellation can’t have a spatial distance > 20% or image size ) Auto by Spatial Referencing 2022/2/15 31

Auto by Spatial Referencing n Matching q q Invariant vectors are computed from the orientation and position of constellations, results are stored in database A candidate image firstly calculate a vector, then try to match the corresponding part in the spatial map stored 2022/2/15 32

Auto by Spatial Referencing n Drawbacks q q q n Still need preprocessing Still need input to be well formed Rejection rate might be very high (no experimental result yet) Advantages q q Very high processing speed, ideally 30 images per second. Increased accuracy 2022/2/15 33

Conclusion n n An survey research of seven representative retina image registration technologies is presented Brief introduction and cons and pros for these methods are provided Vessel structure based methods have a unavoidable limitation that the quality of input retina images will affect the registration results. New methods that can go beyond this, like SURF, would be more promising 2022/2/15 34

Future work n n More representative schemes will be added Registration using Gabor filter q q n Each image has its own orientation, for example, an image with a lot of building has much more horizontal frequency than vertical ones. Garbor filter could be used to help estimating directional blob or direction edge components This is not used in retina image registration yet (or I haven’t read about it). It is also not depend vessel structures, like SURF. If possible, implement some of those schemes 2022/2/15 35

2022/2/15 36