Geometryaware Feature Matching for Structure from Motion Applications

Geometry-aware Feature Matching for Structure from Motion Applications Rajvi Shah, Vanshika Srivastava, P J Narayanan Center for Visual Information Technology IIIT Hyderabad, India

Feature Matching for Sf. M Applications Match features across all image pairs

Pair-wise Feature Matching Image 2 + SIFT Img 2 Descriptors Img 1 Descriptors Image 1 + SIFT

Pair-wise Feature Matching Image 2 + SIFT O(M 2) Exhaustive Comparison Img 2 Descriptors Img 1 Descriptors Image 1 + SIFT

Pair-wise Feature Matching Image 2 + SIFT O(M log M) Kd-tree based Comparison Img 2 Descriptors Img 1 Descriptors Image 1 + SIFT

Kd-tree based Feature Matching Search for top-two near neighbors Ratio of distances between query and 2 -NNs Best Match 2 nd Best Match Q Features in Image 1 Kd-tree of features in Image 2 (in 128 -d descriptor space) Features in Image 2

Kd-tree based Feature Matching Best Match 2 nd Best Match Q Features in Image 1 Kd-tree of features in Image 2 (in 128 -d descriptor space) Features in Image 2

Geometric Verification Distance of the matching point from epipolar line Match Q l = F. Q F-matrix Features in Image 1 l Features in Image 2

Geometric Verification Distance of the matching point from epipolar line d Q Incrorect. Match Distance from l > d l = F. Q F-matrix Features in Image 1 l Features in Image 2

Main Problems § Kd-tree based approach: O(M log M) § Images with features in order on 104 ~ 105. § Wasted effort in matching inconsistent features § Difficult to parallelize § Ratio test is punitive for repetitive structures § Features have similar appearance, e. g. windows. § Salient but not distinctive to pass the ratio test § Rejection of many valid correspondences

Geometry-aware Matching 1. Select a small subset (10%-20%) of features Features in Image 1 Features in Image 2

Geometry-aware Matching 2. Match the selected features using Kd-tree Features in Image 1 Features in Image 2

Geometry-aware Matching 3. Estimate F-Matrix using initial matches (Utilize it for geometry-guided matching) F-matrix Features in Image 1 Features in Image 2

Geometry-aware Matching 4. Find a set of candidate matches Points with Distance from l < d Q l = F. Q l F-matrix Features in Image 1 Features in Image 2

Geometry-aware Matching 5. Find the top 2 -NN within candidates & ratio test Points with Distance from l < d Q l = F. Q l F-matrix Features in Image 1 Features in Image 2

Geometry-aware Matching Candidate set is different for each query! Points with Distance from l < d Q l = F. Q l F-matrix Features in Image 1 Features in Image 2

Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n)

Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. d Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N.

Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. d Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N.

Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. d Optimize Linear Search: O(n) Further Select K points on line l, each at distance d. Radial Search: O(K log n) K << N.

Geometry-aware Matching G 1 Efficiently finding the candidate set Grid based approach: § Divide the image using 4 grids of cell size 2 d x 2 d, overlap d d

Geometry-aware Matching Efficiently finding the candidate set G 2 Grid based approach: § Divide the image using 4 grids of cell size 2 d x 2 d, overlap d d

Geometry-aware Matching Efficiently finding the candidate set Grid based approach: § Divide the image using 4 grids of cell size 2 d x 2 d, overlap d G 3 d

Geometry-aware Matching Efficiently finding the candidate set Grid based approach: G 4 d § Divide the image using 4 grids of cell size 2 d x 2 d, overlap d § Bin the features overlapping cells. into the § Each cell index list of feature indices binned in this cell. § Each (x, y) point 4 cells.

Geometry-aware Matching Efficiently finding the candidate set Grid based approach: For each of the K equidistant points, d § Compute its distance from centers of 4 corresponding cells. § Select the closest cell. § Select all points in this cell as candidates for matching. Candidate Search: O(K)

Geometry-aware Matching Efficiently finding the candidate set Grid based approach: • Candidate Search: O(K) • Matching operations reduced: • From (n log n) to (|c|log |c|) • |c| ≈ • ~ 0. 8% of n for 2 MP image.

Geometry-aware Matching Efficiently finding the candidate set F l l’

Geometry-aware Matching § Fewer candidates, faster search (5 -10 x speed up). § Grid based approach is GPU-friendly. § Selective ratio-test: § Retains valid matches for repetitive elements § Denser and more complete point clouds

Datasets & Experiments § Matching Image Pairs of 4 different Scene type § Indoor, Plaza, Monument, Desk objects § Matches were verified by hand § Match-graph construction + SFM Reconstruction § Barcelona Museum, Tsinghua School, Notre Dame § ~ 90 – 200 images § Reconstruction by bundler

Key Results § CPU : ~ 5 -10 times faster than Kd-tree matching § GPU : ~10 times faster than SIFTGPU matching. § Pairwise Matching : 2 – 5 times more matches § Reconstruction : 1. 5 – 6 times more 3 D points

Qualitative Results Unguided Matching Geometry-aware Matching

Correctly Matched Points Unguided Matching Geometry-aware Matching

Qualitative Results 119/191 images 181/191 images Unguided Matching Geometry-aware Matching

Thank You rajvi. shah@research. iiit. ac. in http: //cvit. iiit. ac. in/multistagesfm/