Stanford CS 223 B Computer Vision Winter 2005
Stanford CS 223 B Computer Vision, Winter 2005 Lecture 4 Advanced Features Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp, Stanford [with slides by D Lowe, M. Polleyfeys] Sebastian Thrun CS 223 B Computer Vision, Winter 2005 1
Today’s Goals • • Harris Corner Detector Hough Transform Templates, Image Pyramid, Transforms SIFT Features Sebastian Thrun CS 223 B Computer Vision, Winter 2005 2
Finding Corners Edge detectors perform poorly at corners. Corners provide repeatable points for matching, so are worth detecting. Idea: • Exactly at a corner, gradient is ill defined. • However, in the region around a corner, gradient has two or more different values. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 3
The Harris corner detector Form the second-moment matrix: Sum over a small region around the hypothetical corner Matrix is symmetric Sebastian Thrun Gradient with respect to x, times gradient with respect to y Slide credit: David Jacobs CS 223 B Computer Vision, Winter 2005 4
Simple Case First, consider case where: This means dominant gradient directions align with x or y axis If either λ is close to 0, then this is not a corner, so look for locations where both are large. Slide credit: David Jacobs Sebastian Thrun CS 223 B Computer Vision, Winter 2005 5
General Case It can be shown that since C is symmetric: So every case is like a rotated version of the on last slide. Slide credit: David Jacobs Sebastian Thrun CS 223 B Computer Vision, Winter 2005 6
So, To Detect Corners • Filter image with Gaussian to reduce noise • Compute magnitude of the x and y gradients at each pixel • Construct C in a window around each pixel (Harris uses a Gaussian window – just blur) • Solve for product of ls (determinant of C) • If ls are both big (product reaches local maximum and is above threshold), we have a corner (Harris also checks that ratio of ls is not too high) Sebastian Thrun CS 223 B Computer Vision, Winter 2005 7
Gradient Orientation Closeup Sebastian Thrun CS 223 B Computer Vision, Winter 2005 8
Corner Detection Corners are detected where the product of the ellipse axis lengths reaches a local maximum. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 9
Harris Corners • Originally developed as features for motion tracking • Greatly reduces amount of computation compared to tracking every pixel • Translation and rotation invariant (but not scale invariant) Sebastian Thrun CS 223 B Computer Vision, Winter 2005 10
Harris Corner in Matlab % Harris Corner detector - by Kashif Shahzad sigma=2; thresh=0. 1; sze=11; disp=0; % Derivative masks dy = [-1 0 1; -1 0 1]; dx = dy'; %dx is the transpose matrix of dy % Ix and Iy are the horizontal and vertical edges of image Ix = conv 2(bw, dx, 'same'); Iy = conv 2(bw, dy, 'same'); % Calculating the gradient of the image Ix and Iy g = fspecial('gaussian', max(1, fix(6*sigma)), sigma); Ix 2 = conv 2(Ix. ^2, g, 'same'); % Smoothed squared image derivatives Iy 2 = conv 2(Iy. ^2, g, 'same'); Ixy = conv 2(Ix. *Iy, g, 'same'); % My preferred measure according to research paper cornerness = (Ix 2. *Iy 2 - Ixy. ^2). /(Ix 2 + Iy 2 + eps); % We should perform nonmaximal suppression and threshold mx = ordfilt 2(cornerness, sze^2, ones(sze)); % Grey-scale dilate cornerness = (cornerness==mx)&(cornerness>thresh); % Find maxima [rws, cols] = find(cornerness); % Find row, col coords. clf ; imshow(bw); hold on; p=[cols rws]; plot(p(: , 1), p(: , 2), 'or'); title('bf Harris Corners') Sebastian Thrun CS 223 B Computer Vision, Winter 2005 11
Today’s Goals • • Harris Corner Detector Hough Transform Templates, Image Pyramid, Transforms SIFT Features Sebastian Thrun CS 223 B Computer Vision, Winter 2005 12
Features? Local versus global Sebastian Thrun CS 223 B Computer Vision, Winter 2005 13
Vanishing Points Sebastian Thrun CS 223 B Computer Vision, Winter 2005 14
Vanishing Points A. Canaletto [1740], Arrival of the French Ambassador in Venice Sebastian Thrun CS 223 B Computer Vision, Winter 2005 15
From Edges to Lines Sebastian Thrun CS 223 B Computer Vision, Winter 2005 16
Hough Transform Sebastian Thrun CS 223 B Computer Vision, Winter 2005 17
Hough Transform: Quantization m m Detecting Lines by finding maxima / clustering in parameter space Sebastian Thrun CS 223 B Computer Vision, Winter 2005 18
Hough Transform: Results Image Sebastian Thrun Edge detection Hough Transform CS 223 B Computer Vision, Winter 2005 19
Today’s Goals • • Harris Corner Detector Hough Transform Templates, Image Pyramid, Transforms SIFT Features Sebastian Thrun CS 223 B Computer Vision, Winter 2005 20
Problem: Features for Recognition Want to find Sebastian Thrun … in here CS 223 B Computer Vision, Winter 2005 21
Convolution with Templates % read image im = imread('bridge. jpg'); bw = double(im(: , 1)). / 256; ; imshow(bw) % select an image patch = bw(221: 240, 351: 370); imshow(patch) patch = patch - (sum(patch)) / size(patch, 1) / size(patch, 2)); % apply FFTim = fft 2(bw); bw 2 = real(ifft 2(FFTim)); imshow(bw 2) kernel=zeros(size(bw)); kernel(1: size(patch, 1), 1: size(patch, 2)) = patch; FFTkernel = fft 2(kernel); % define a kernel=zeros(size(bw)); kernel(1, 1) = 1; kernel(1, 2) = -1; FFTkernel = fft 2(kernel); % apply the kernel and check out the result FFTresult = FFTim. * FFTkernel; result = max(0, real(ifft 2(FFTresult))); result = result. / max(result)); result = (result. ^ 1 > 0. 5); % apply the kernel and check out the imshow(result) result FFTresult = FFTim. * FFTkernel; % alternative convolution result = real(ifft 2(FFTresult)); imshow(conv 2(bw, patch, 'same')) imshow(result) Sebastian Thrun CS 223 B Computer Vision, Winter 2005 22
Convolution with Templates • Invariances: – Scaling – Rotation – Illumination – Deformation No No No Maybe • Provides – Good localization Sebastian Thrun No CS 223 B Computer Vision, Winter 2005 23
Scale Invariance: Image Pyramid Sebastian Thrun CS 223 B Computer Vision, Winter 2005 24
Aliasing Sebastian Thrun CS 223 B Computer Vision, Winter 2005 25
Aliasing Effects Constructing a pyramid by taking every second pixel leads to layers that badly misrepresent the top layer Sebastian Thrun CS 223 B Computer Vision, Winter 2005 26
Convolution Theorem Fourier Transform of g: F is invertible Sebastian Thrun CS 223 B Computer Vision, Winter 2005 27
The Fourier Transform • Represent function on a new basis – Think of functions as vectors, with many components – We now apply a linear transformation to transform the basis • dot product with each basis element • In the expression, u and v select the basis element, so a function of x and y becomes a function of u and v • basis elements have the form Sebastian Thrun CS 223 B Computer Vision, Winter 2005 28
Fourier basis element example, real part Fu, v(x, y)=const. for (ux+vy)=const. Vector (u, v) • Magnitude gives frequency • Direction gives orientation. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 29
Here u and v are larger than in the previous slide. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 30
And larger still. . . Sebastian Thrun CS 223 B Computer Vision, Winter 2005 31
Phase and Magnitude • Fourier transform of a real function is complex – difficult to plot, visualize – instead, we can think of the phase and magnitude of the transform • Phase is the phase of the complex transform • Magnitude is the magnitude of the complex transform • Curious fact – all natural images have about the same magnitude transform – hence, phase seems to matter, but magnitude largely doesn’t • Demonstration – Take two pictures, swap the phase transforms, compute the inverse - what does the result look like? Sebastian Thrun CS 223 B Computer Vision, Winter 2005 32
Sebastian Thrun CS 223 B Computer Vision, Winter 2005 33
This is the magnitude transform of the cheetah picture Sebastian Thrun CS 223 B Computer Vision, Winter 2005 34
This is the phase transform of the cheetah picture Sebastian Thrun CS 223 B Computer Vision, Winter 2005 35
Sebastian Thrun CS 223 B Computer Vision, Winter 2005 36
This is the magnitude transform of the zebra picture Sebastian Thrun CS 223 B Computer Vision, Winter 2005 37
This is the phase transform of the zebra picture Sebastian Thrun CS 223 B Computer Vision, Winter 2005 38
Reconstruction with zebra phase, cheetah magnitude Sebastian Thrun CS 223 B Computer Vision, Winter 2005 39
Reconstruction with cheetah phase, zebra magnitude Sebastian Thrun CS 223 B Computer Vision, Winter 2005 40
Convolution with Templates • Invariances: – Scaling – Rotation – Illumination – Deformation Yes No No Maybe • Provides – Good localization Sebastian Thrun No CS 223 B Computer Vision, Winter 2005 41
Today’s Goals • • Harris Corner Detector Hough Transform Templates, Image Pyramid, Transforms SIFT Features Sebastian Thrun CS 223 B Computer Vision, Winter 2005 42
Let’s Return to this Problem… Want to find Sebastian Thrun … in here CS 223 B Computer Vision, Winter 2005 43
SIFT • Invariances: – Scaling – Rotation – Illumination – Deformation Yes Yes Maybe • Provides – Good localization Sebastian Thrun Yes CS 223 B Computer Vision, Winter 2005 44
SIFT Reference Distinctive image features from scale-invariant keypoints. David G. Lowe, International Journal of Computer Vision, 60, 2 (2004), pp. 91 -110. SIFT = Scale Invariant Feature Transform Sebastian Thrun CS 223 B Computer Vision, Winter 2005 45
Invariant Local Features • Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters SIFT Features Sebastian Thrun CS 223 B Computer Vision, Winter 2005 46
Advantages of invariant local features • Locality: features are local, so robust to occlusion and clutter (no prior segmentation) • Distinctiveness: individual features can be matched to a large database of objects • Quantity: many features can be generated for even small objects • Efficiency: close to real-time performance • Extensibility: can easily be extended to wide range of differing feature types, with each adding robustness 47
SIFT On-A-Slide 1. 2. 3. 4. 5. 6. Enforce invariance to scale: Compute Gaussian difference max, for may different scales; non-maximum suppression, find local maxima: keypoint candidates Localizable corner: For each maximum fit quadratic function. Compute center with sub-pixel accuracy by setting first derivative to zero. Eliminate edges: Compute ratio of eigenvalues, drop keypoints for which this ratio is larger than a threshold. Enforce invariance to orientation: Compute orientation, to achieve scale invariance, by finding the strongest second derivative direction in the smoothed image (possibly multiple orientations). Rotate patch so that orientation points up. Compute feature signature: Compute a "gradient histogram" of the local image region in a 4 x 4 pixel region. Do this for 4 x 4 regions of that size. Orient so that largest gradient points up (possibly multiple solutions). Result: feature vector with 128 values (15 fields, 8 gradients). Enforce invariance to illumination change and camera saturation: Normalize to unit length to increase invariance to illumination. Then threshold all gradients, to become invariant to camera saturation. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 48
SIFT On-A-Slide 1. 2. 3. 4. 5. 6. Enforce invariance to scale: Compute Gaussian difference max, for may different scales; non-maximum suppression, find local maxima: keypoint candidates Localizable corner: For each maximum fit quadratic function. Compute center with sub-pixel accuracy by setting first derivative to zero. Eliminate edges: Compute ratio of eigenvalues, drop keypoints for which this ratio is larger than a threshold. Enforce invariance to orientation: Compute orientation, to achieve scale invariance, by finding the strongest second derivative direction in the smoothed image (possibly multiple orientations). Rotate patch so that orientation points up. Compute feature signature: Compute a "gradient histogram" of the local image region in a 4 x 4 pixel region. Do this for 4 x 4 regions of that size. Orient so that largest gradient points up (possibly multiple solutions). Result: feature vector with 128 values (15 fields, 8 gradients). Enforce invariance to illumination change and camera saturation: Normalize to unit length to increase invariance to illumination. Then threshold all gradients, to become invariant to camera saturation. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 49
Find Invariant Corners 1. Enforce invariance to scale: Compute Gaussian difference max, for may different scales; non-maximum suppression, find local maxima: keypoint candidates Sebastian Thrun CS 223 B Computer Vision, Winter 2005 50
Finding “Keypoints” (Corners) Idea: Find Corners, but scale invariance Approach: • Run linear filter (diff of Gaussians) • At different resolutions of image pyramid Sebastian Thrun CS 223 B Computer Vision, Winter 2005 51
Difference of Gaussians Minus Equals Sebastian Thrun CS 223 B Computer Vision, Winter 2005 52
Difference of Gaussians surf(fspecial('gaussian', 40, 4)) surf(fspecial('gaussian', 40, 8) - fspecial('gaussian', 40, 4)) Sebastian Thrun CS 223 B Computer Vision, Winter 2005 53
Find Corners with Diff. Of. Gauss im =imread('bridge. jpg'); bw = double(im(: , 1)) / 256; for i = 1 : 10 gauss. D = fspecial('gaussian', 40, 2*i) - fspecial('gaussian', 40, i); % mesh(gauss. D) ; drawnow res = abs(conv 2(bw, gauss. D, 'same')); res = res / max(res)); imshow(res) ; title(['bf i = ' num 2 str(i)]); drawnow end Sebastian Thrun CS 223 B Computer Vision, Winter 2005 54
Build Scale-Space Pyramid • All scales must be examined to identify scale-invariant features • An efficient function is to compute the Difference of Gaussian (DOG) pyramid (Burt & Adelson, 1983) Sebastian Thrun CS 223 B Computer Vision, Winter 2005 55
Key point localization • Detect maxima and minima of difference-of-Gaussian in scale space Sebastian Thrun CS 223 B Computer Vision, Winter 2005 56
Example of keypoint detection (a) 233 x 189 image (b) 832 DOG extrema Sebastian Thrun CS 223 B Computer Vision, Winter 2005 57
SIFT On-A-Slide 1. 2. 3. 4. 5. 6. Enforce invariance to scale: Compute Gaussian difference max, for may different scales; non-maximum suppression, find local maxima: keypoint candidates Localizable corner: For each maximum fit quadratic function. Compute center with sub-pixel accuracy by setting first derivative to zero. Eliminate edges: Compute ratio of eigenvalues, drop keypoints for which this ratio is larger than a threshold. Enforce invariance to orientation: Compute orientation, to achieve scale invariance, by finding the strongest second derivative direction in the smoothed image (possibly multiple orientations). Rotate patch so that orientation points up. Compute feature signature: Compute a "gradient histogram" of the local image region in a 4 x 4 pixel region. Do this for 4 x 4 regions of that size. Orient so that largest gradient points up (possibly multiple solutions). Result: feature vector with 128 values (15 fields, 8 gradients). Enforce invariance to illumination change and camera saturation: Normalize to unit length to increase invariance to illumination. Then threshold all gradients, to become invariant to camera saturation. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 58
Example of keypoint detection Threshold on value at DOG peak and on ratio of principle curvatures (Harris approach) (c) 729 left after peak value threshold (d) 536 left after testing ratio of principle curvatures Sebastian Thrun CS 223 B Computer Vision, Winter 2005 59
SIFT On-A-Slide 1. 2. 3. 4. 5. 6. Enforce invariance to scale: Compute Gaussian difference max, for may different scales; non-maximum suppression, find local maxima: keypoint candidates Localizable corner: For each maximum fit quadratic function. Compute center with sub-pixel accuracy by setting first derivative to zero. Eliminate edges: Compute ratio of eigenvalues, drop keypoints for which this ratio is larger than a threshold. Enforce invariance to orientation: Compute orientation, to achieve scale invariance, by finding the strongest second derivative direction in the smoothed image (possibly multiple orientations). Rotate patch so that orientation points up. Compute feature signature: Compute a "gradient histogram" of the local image region in a 4 x 4 pixel region. Do this for 4 x 4 regions of that size. Orient so that largest gradient points up (possibly multiple solutions). Result: feature vector with 128 values (15 fields, 8 gradients). Enforce invariance to illumination change and camera saturation: Normalize to unit length to increase invariance to illumination. Then threshold all gradients, to become invariant to camera saturation. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 60
Select canonical orientation • Create histogram of local gradient directions computed at selected scale • Assign canonical orientation at peak of smoothed histogram • Each key specifies stable 2 D coordinates (x, y, scale, orientation) Sebastian Thrun CS 223 B Computer Vision, Winter 2005 61
SIFT On-A-Slide 1. 2. 3. 4. 5. 6. Enforce invariance to scale: Compute Gaussian difference max, for may different scales; non-maximum suppression, find local maxima: keypoint candidates Localizable corner: For each maximum fit quadratic function. Compute center with sub-pixel accuracy by setting first derivative to zero. Eliminate edges: Compute ratio of eigenvalues, drop keypoints for which this ratio is larger than a threshold. Enforce invariance to orientation: Compute orientation, to achieve scale invariance, by finding the strongest second derivative direction in the smoothed image (possibly multiple orientations). Rotate patch so that orientation points up. Compute feature signature: Compute a "gradient histogram" of the local image region in a 4 x 4 pixel region. Do this for 4 x 4 regions of that size. Orient so that largest gradient points up (possibly multiple solutions). Result: feature vector with 128 values (15 fields, 8 gradients). Enforce invariance to illumination change and camera saturation: Normalize to unit length to increase invariance to illumination. Then threshold all gradients, to become invariant to camera saturation. Sebastian Thrun CS 223 B Computer Vision, Winter 2005 62
SIFT vector formation • Thresholded image gradients are sampled over 16 x 16 array of locations in scale space • Create array of orientation histograms • 8 orientations x 4 x 4 histogram array = 128 dimensions Sebastian Thrun CS 223 B Computer Vision, Winter 2005 63
Nearest-neighbor matching to feature database • Hypotheses are generated by approximate nearest neighbor matching of each feature to vectors in the database – SIFT use best-bin-first (Beis & Lowe, 97) modification to k-d tree algorithm – Use heap data structure to identify bins in order by their distance from query point • Result: Can give speedup by factor of 1000 while finding nearest neighbor (of interest) 95% of the time Sebastian Thrun CS 223 B Computer Vision, Winter 2005 64
3 D Object Recognition • Extract outlines with background subtraction Sebastian Thrun CS 223 B Computer Vision, Winter 2005 65
3 D Object Recognition • Only 3 keys are needed for recognition, so extra keys provide robustness • Affine model is no longer as accurate Sebastian Thrun CS 223 B Computer Vision, Winter 2005 66
Recognition under occlusion Sebastian Thrun CS 223 B Computer Vision, Winter 2005 67
Test of illumination invariance • Same image under differing illumination 273 keys verified in final match Sebastian Thrun CS 223 B Computer Vision, Winter 2005 68
Examples of view interpolation Sebastian Thrun CS 223 B Computer Vision, Winter 2005 69
Location recognition Sebastian Thrun CS 223 B Computer Vision, Winter 2005 70
Sony Aibo (Evolution Robotics) SIFT usage: Recognize charging station Communicate with visual cards Sebastian Thrun CS 223 B Computer Vision, Winter 2005 71
Today’s Goals • • Harris Corner Detector Hough Transform Templates, Image Pyramid, Transforms SIFT Features Sebastian Thrun CS 223 B Computer Vision, Winter 2005 72
- Slides: 72