Evaluation of the stability of SIFT keypoint correspondence

Evaluation of the stability of SIFT keypoint correspondence across cameras or. . “can we put a ‘C’ in SIFT? ” max van kleek 6. 869: learning and interfaces thursday may 11, 2005

ubiquitous computing: computers (and cameras) are everywhere!

little sister: follow-me-around user modeling object correspondence across varying cameras/lighting/scenes is an important subgoal

applications: buliding a personal life log for yourself, interest profiling, social network mining, health care

identifying objects with local features: the SIFT transform

Orientation histogram = SIFT feature vector

identified and oriented keypoints used to vote for pose orientations in a hough transform pose-and-scale space recognizing objects

keypoint correspondence 1. % keypoints detected 2. stability of orientation histogram object orientation (away from frontal parallel) Lowe, D. G. “Distinctive Image Features from Scale-Invariant Keypoints”, ICJV 2004. object deformation ? ? ? lighting direction, intensity, shading Mikolajczyk, K. , C. Schmidt “A Performance Evaluation of Local Descriptors”, CVPR ‘ 03 lens distortion, sharpness, ccd “quality”, noise, capture artifacts Me! well, sort of. .

the experiment cameras vary widely in sizes, configurations, capabilities, and prices Logitech QC Express Logitech QC Pro 3000 640 x 480 16 -bit color YUV 4: 2: 2 AGC, Auto Exposure manual focus USB iface $15 CCD by Phillips 640 x 480 16 -bit color YUV 4: 2: 2, RGB AGC, Auto Exposure Auto WB manual focus USB iface $50 Sony EVI-D 30 Steerable NTSC camera, DV capture card 720 x 480 luminance, less for color Raw DV AGC, Auto Exposure Auto focus Nikon Coolpix 990 Digital still camera, 2048 x 1536 RGB Auto Gain, Auto WB, Auto Exposure Auto focus $1000 -> $500 ~$300 + $200

rim ent nde set nsc up: twe ent en l cam ights era and sub j ect

acquiring image sets for each camera: background 10 images stationary 2 front 2 face right 2 face left = 16 images/cam * 4 cameras = 48

320 x 240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100; default camera settings except disabled AGC, disabled AE (locked to optimal settings)

contrast-stretched background images algorithm for keypoint correspondence background model (mean) source image contrast stretch over whole set compute_sift_points find(p(img) < epsilon)

dilate fg mask with a disc strel keeping only relevant keypoints by intersecting keypoints with foreground pts intersection filter out bg key points

image A w/ sift keypoints orientation histograms for each keypoint in A image B w/ sift keypoints orientation histograms for each keypoint in B

match keypoints using nearest-neighbor in SIFT space orientation histograms for each keypoint in A orientation histograms for each keypoint in B

match keypoints using nearest-neighbor in SIFT space orientation histograms for each keypoint in A orientation histograms for each keypoint in B

sanity check: same (dv) camera, slightly different pose 1. 1 -> 1. 2 1. 1: 15 keypoints detected 1. 2: 11 keypoints detected 1. 2 -> 1. 1 6� properly assigned 8 in common 5 properly assigned 8 in common

4. 1: 18 keypoints detected 1. 1: 15 keypoints detected nikon coolpix versus sony steerable 1. 1 -> 4. 1 -> 1. 1 5� properly assigned 10 in common

4. 1: 18 keypoints detected 3. 1: 17 keypoints detected nikon coolpix versus qc pro 4. 1 -> 3. 1 -> 4. 1 2 properly assigned 10 in common 5� properly assigned 10 in common

4. 1: 18 keypoints detected 2. 1: 22 keypoints detected nikon coolpix versus qc express 4. 1 -> 2. 1 2 properly assigned 10 in common�� 0� properly assigned 10 in common

other results: qc pro vs qc exp (3. 1 -> 2. 1) 3. 1 ->2. 1 : 17 / 22 4 correct out of 6 in common 2. 1 ->3. 1: 22 / 17 0 correct out of 6 in common poor reproducibility with qcs? qc exp test (is the qc exp just too noisy? ) 2. 1 ->2. 2 : 22/20 1 correct out of 8 in common 2. 2 ->2. 1 : 20/22 2 correct out of 8 in common yes. qc pro reproducibility 3. 1 ->3. 2: 17/24 6 correct out of 9 in common 3. 2 ->3. 1 : 24/17 7 correct out of 9 in common angle test using qc pro 3. 1 ->3. 4 : 0 correct out of 0 in common sensitive to out-of-plane rotation

experiment setup: 5 incandenscent lights 3 ft between camera and robot

320 x 240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100; default camera settings except disabled AGC, (locked to optimal settings)

• parameters: – – bins / histogram pixels / quadrants / keypoint gaussian dropoff covariance keypoint splitting / multiple primary gradient directions source keypoint merging histogram blurring