Invariant Local Feature for Object Recognition Presented by

Invariant Local Feature for Object Recognition Presented by Wyman 2/05/2006 1

Introduction n Object Recognition q q A task of finding 3 D objects from 2 D images (or even video) and classifying them into one of the many known object types Closely related to the success of many computer vision applications n q robotics, surveillance, registration … etc. A difficult problem that a general and comprehensive solution to this problem has not been made 2

Introduction n Two main streams of approaches: q q Model-Based Object Recognition View-Based Object Recognition n n 2 D representations of the same object viewed at different angles and distances when available Extract features (as the representations of object) and compare them to those in the feature database 3

Matching with Local Features n One of the possible solution q Matching with invariant local features n n Robust to Occlusion, clutter background cf. global features Repeatedly Detected n Three phases: q q q Detection Description Matching Accurate, Fast Distinctive Invariance 4

Research Direction n Study and improve the invariant local features q Detection, description and matching n Study and improve object recognition / matching using invariant local features n Area to improve q q q Distinctiveness Invariance Efficiency 5

Outline n State-of-the-art techniques q q n Descriptor Matching Conclusion & Future Works 6

Outline n State-of-the-art techniques q Descriptor n n n q n Performance evaluation Current extension using color Possible way to improve – Color Orientation Matching Conclusion & Future Work 7

Outline n State-of-the-art techniques q Descriptor n n n q Matching n n Performance evaluation Current extension using color Possible way to improve – Color Orientation Cross-bin distance Performance evaluation Possible way to improve – Aggregation of Content Conclusion & Future Work 8

Performance Evaluation of Descriptors n We aim to compare the performance of three state-of-the-art local feature descriptors: SIFT, PCA-SIFT and GLOH n Same experimental setup as that used in “Performance Evaluation of Local Descriptors” TPAMI 2005 q q n Different evaluation criterion Different result In each experiment, each descriptor describe features from q q Harris corner detector Harris-affine covariant detector n Output regions that are invariant to viewpoint change 9

SIFT – Scale Invariant Feature Transform Invariance n Detector Descriptor Scale Rotation Illumination Viewpoint Descriptor overview: q q q Find local orientation as the dominant gradient direction Rotation Invariant Compute gradient orientation histograms of several small windows (128 values for each point) relative to the local orientation Viewpoint Invariant Normalize the descriptor to make it invariant to intensity change Illumination D. Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004 10

PCA-SIFT n n n Rotate feature region to dominant gradient direction same as SIFT Pre-compute an eigenspace for local gradient patches of size 41 x 41 2 x 39=3042 elements Only keep 20 components A more compact descriptor Sensitive to viewpoint change 11 Y. K. Rahul. Pca-sift: A more distinctive representation for local image descriptors. CVPR 2004

GLOH (Gradient location-orientation histogram) n Different from SIFT in sampling method q q n n 17 log-polar location bins 16 orientation bins PCA on Orientation Histogram VS PCA on Gradient Patch Analyze the 17 x 16=272 Dimensions Apply PCA analysis, keep 128 components 17 Log-polar location bins 12 C. S. Krystian Mikolajczyk. A performance evaluation of local descriptors. TPAMI 2005

Performance Evaluation Scale + Rotation (bark) n Data Set q From Visual Geometry Group Viewpoint change (graf) Illumination change (leuven) Blur (bikes) Blurring Viewpoint change (wall) 13

Performance Evaluation n Evaluation Criteria q Match features from first image to the second one based on the nearest neighbor distance ratio n n q q That is, two features are matched if first nearest neighbor is much closer than the second nearest neighbor This is different from the threshold-based criterion used in “A Performance Evaluation of Local Descriptors” TPAMI 2005 Count the number of correct matches and the number of false matches obtained for an image pair The results are plotted in form of recall versus 1 -precision curves Total # possible matches 14

Performance Evaluation Viewpoint change (wall) Scale + Rotation (bark) Viewpoint change (graf) Blurring (bikes) Illumination change (leuven) 15

Performance Evaluation Result n n n Descriptor Distinctiveness Complexity Feature Size SIFT High Medium 128 PCA-SIFT Medium Low 20 GLOH High 128 For accuracy SIFT For speed PCA-SIFT In large database ? 16

Start from Scratch n Comparison of my descriptor with SIFT q n Simply designed vs carefully designed Result q SIFT is a carefully Increasing illumination change designed descriptor, it remains robust when the degree of transformation increases Increasing affine change Increasing blur 17

Extension using Color n Weijier extends local feature descriptors with color information, by concatenating a color descriptor, K, to the shape descriptor, S, according to n where B is the combined color and shape descriptor and is a weighting parameter and ^ indicates that the vector is normalized. J. van de Weijer and C. Schmid. Coloring local feature extraction. ECCV 2006. 18

Proposed Extension using Color n Problem statement q Orientation of local feature patch are obtained from the monochrome intensity image … q q Color feature patches on the right has the same grayscale patches shown on the left. Thus, they are assigned the same orientation histogram If we can generate significant orientation histogram for each of them, we can further improve the distinctiveness of the shape descriptor, SIFT 19

Feature Matching n n Original distance metric designed for SIFT, PCA-SIFT and GLOH is bin-to-bin Euclidean distance Problems: q q Sensitive to quantization effects Sensitive to distortion problems due to deformation, illumination change and noise 20

Feature Matching – Diffusion Distance n Haibin Ling proposed a new distance metric for histogrambased descriptor called diffusion distance Gaussian Blur In 1 direction 1 D case n Gaussian Blur In 3 directions 3 D case Summing value in all layers of the distance pyramid with exponentially decreasing size H. Ling and K. Okada. Diffusion distance for histogram comparison. CVPR 06. 21

Feature Matching – Performance Evaluation n n Same setup as the previous experiment Recall vs 1 -prevision curve for image pair with affine transformation 22

Feature Matching – Performance Evaluation Data set. The synthetic deformation data set from Haibin Ling Images in the data set and the evaluation method needs to be improved 23

Proposed Extension n n Robust aggregation of the histogram, such as average orientation direction and center of mass of derivatives, can be also used in comparison Diffusion distance can be viewed as a form of comparison using the aggregate information Its aggregation of histogram bins is obtained by repeatedly convolving the histogram with Gaussian kernels q Summation of the distance between each aggregation pair of two histograms gives the diffusion distance Histogram A Histogram B q 128 bins 64 bins 32 bins Aggregation: 1. Average of gradient magnitude over location bins 2. Bin reduction in orientation bins 24

Conclusion and Future Work n Presented q q n Result of performance evaluation of some state-of-the-art descriptors and feature matching distance metric Possible way to improve the description and matching step TODO q Incorporate color information into local features n q Improve feature’s distinctiveness Design a distance metric for comparing SIFT feature’s histogram n n Invariant to deformation (like diffusion distance) Improve feature’s distinctiveness 25

Q&A Thank you very much! 26

Models of Image Change n Geometry q q q Rotation Similarity (rotation + uniform scale) Affine (scale dependent on direction) valid for: orthographic camera, locally planar object n Photometry q Affine intensity change (I a I + b) 27