Feature Detection and Matching in Images with Radial

Feature Detection and Matching in Images with Radial Distortion Miguel Lourenço, João P. Barreto, Abed Malti Institute for Systems and Robotics, Faculty of Science and Technology University of Coimbra, Portugal

Presentation Outline SIFT Features – brief overview RD problems in keypoint detection and matching § Theoretical reasoning § Experimental validation Improvement to the SIFT algorithm to enhance it with RD Real experiments – a comparison study Motion estimation and 3 D reconstruction in endoscopic images Name / Location / Date Slide 2

Motivation for keypoint detection and matching Point correspondence across multiple views § § Camera calibration Sparse 3 D reconstruction Recover camera/robot motion Visual Slam Representation of image content § Image retrieval applications § Recognition tasks (e. g. Voc-tree) § Image compression Partioning of the descriptor space Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 3

SIFT Features (Lowe, IJCV 2004 – 6725 citations on google scholar) SIFT is probably the most broadly used algorithm for keypoint detection and matching How does SIFT work ? § Image salient points detected in a scale space framework Increase scale (x, y, sigma) Gaussian pyramid Do. G pyramid Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 4

SIFT Features (Lowe, IJCV 2004 – 6725 citations on Scholar google) SIFT is probably the most broadly used algorithm for keypoint detection and matching How does SIFT work ? § Image salient points detected in a scale space framework § SIFT descriptor is computed based on local image gradient on a scale and rotation normalized patch Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 5

Problem statement (1/2) What is radial distortion? § Bending of the light rays pulling image points towards the center along radial direction Q X O' O Z Cameras with radial lens distortion are often used in computer and robotic vision applications Mini-lens Boroscope Fish-eye lens Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 6

Problem statement (2/2) SIFT is invariant to rotation and scale but it is not invariant to 421 correct matches 336 correct matches RD Our Contribution: Modifications to the original SIFT for invariance to image RD Assumptions: § RD can be fairly described by the division model (Fitzgibbon, CVPR 2001) § RD is roughly known ( e. g. line stretching ) (Barreto, CVIU 2006) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 7

Tracking RD effects in SIFT How does RD affect the SIFT algorithm? § Study using images with artificially added distortion Ä Isolate the RD effect in SIFT detection and matching Ä Reliable ground truth RD = 0% RD = 15% RD = 35% RD = 55% Improvement to the SIFT algorithm to handle RD issues Results on real imagery Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 8

How does RD affect keypoint detection? Repeatability of keypoint detection decreases with increasing distortion Filtering bounds § Small features (fine scale) tend to disappear during the blurring process § Coarse features tend to be detected at finer levels of scale § Flat regions (e. g. edges) start gain to strong gradient variations Regular Do. G pyramid ‘Distorted’ Do. G pyramid Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 9

Proposed Solution: Adaptive smoothing Radial distortion must be removed before the Gaussian smoothing Rectification (~ 1. 5 seconds in Matlab) We can avoid the reconstruction artifacts by using an adaptive filter Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 10

Standard vs Adaptive Gaussian smoothing Inherent properties of the standard Gaussian filter § Decouple the convolution mask in X and Y directions Simplification of the adaptive filter Advantages of the Simplified Adaptive Filter § Shape only depends on the radius of the convolution window § Isotropic filter that can be decoupled for each image radius Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 11

Detection repeatability (synthetic adding of RD) Repeatability Computational time Error in calibration ü Better Repeatability results for keypoint detection ü Lower computational time than image rectification ü More robust to calibration errors Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 12

How does RD affect matching? RD modifies the local structures in the image and by consequence the gradients are affected Changes in local gradients of the image deteriorates SIFT descriptor performance Proposed solution: Compute gradients in the distorted image and perform implicit correction using the jacobian matrix of the distortion function Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 13

Matching evaluation (synthetic adding of RD) Implicit gradient correction SIFT in Rectified Images SIFT in RD Images ü Compressive effect adds new contributions to the descriptor that do not occur in undistorted views ü The matching performance can be improved by correcting image gradients before building the descriptor ü Implicit gradient correction outperforms explicit image rectification for distortion amounts up to 25%. Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 14

Experiments with Real Images Planar scenes for repeatability test and scenes with depth variation for motion estimation § Firewire camera with regular lens (~ 10% ) of distortion § Dragonfly camera with mini lens (~ 25% ) of distortion § Firewire camera with fish-eye lens (~ 45% ) of distortion Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 15

Planar images SIFT in RD Images SIFT in Rectified Images Our method 176 matches 294 matches 364 matches 201 matches 310 matches 401 matches 112 matches 253 matches 326 matches 10% distortion 25% distortion 45% distortion Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 16

Motion recovery / Sparse 3 D reconstruction Scenes with depth variation where wrong matches are discarded using epipolar geometric constraints Main Scene 10% distortion 25% distortion 45% distortion Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 17

Experimental evaluation Main Scene Number of Inliers RMS rotation angle 3 D reconstruction Inliers Distribution Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 18

Conclusions / Future Work We proposed a set of modifications to the original SIFT algorithm (RD-SIFT) for achieving invariance to radial distortion. The additional computational overhead is minimum RD – SIFT proved to be superior to explicit image correction § Better repeatability and retrieval performance § Less computational overhead § Increased robustness to calibration errors Future Work § Extend the approach to other keypoint detectors (e. g. MSER and SURF) § Real-time implementation using GPGPU (to make available to the community) § Get rid of calibration dependence Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 19

Name / Location / Date Slide 20

THANKS FOR COMING Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 21

Detection after explicit RD correction Correct the radial distortion via image rectification Rectification (~ 1. 5 seconds in Matlab) Drawbacks of this approach § Signal reconstruction introduces artifacts affecting SIFT performance Image 1. 5 x Image (Bilinear) 1. 5 x Image (Bicubic) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 22