Corner Detection The Correspondence problem Digital Image Processing
Corner Detection The Correspondence problem Digital Image Processing Lecture 9 Martin Urban Center for Machine Perception FEE, CTU Prague (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem
Lecture Overview 1. Intro to “The Correspondence Problem” - definition - demo of typical applications - Corner Detection helps to solve C. P. 2. Corner Detection – Harris Algorithm 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 2
What it is “The correspondence problem”? A B Ex. result: x-shift “Ideal” solution: 9/10/2020 a pixel to pixel mapping from A to B (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 3
Applications: 3 D Reconstruction 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 4
3 D Reconstruction 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 5
Camera motion tracking ⇒ image stabilization original 9/10/2020 stabilized (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 6
Camera motion tracking ⇒ 3 D animation 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 7
Motion keying / segmentation input sequence Background sequence 9/10/2020 Foreground sequence (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 8
Motion Detection Input: Output: 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 9
Medical imaging – image registration from the atlas before registration after test slice 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem deform. field 10
The correspondence problem A B “Ideal” solution: a pixel to pixel mapping from A to B How to do it? 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 11
The correspondence problem A Intuitive approach: For ∀ pxl ∈ A, find a pxl ∈ B with the most similar neighbourhood. B Problems: - How to measure similarity of image patches? - undistinguishable regions (e. g. texture-less) - not surjective map (onto) due to occlusions - not bijective map (one to one) due to scale changes - huge data 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem ⇒ very hard / impossible to recover the mapping 12
The correspondence problem Conclusion: It is very hard / impossible to recover dense pxl 2 pxl mapping between two images. Solution: 1. Recover the correspondence relation just between several well distinguished image features (interest points / corners / Harris points). 2. Estimate multi-view transformation (e. g. epipolar geometry, camera motion). 3. Having epipolar geometry, try to find dense correspondences. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 13
Corner Detection: Introduction undistinguished patches: Corner detector detects points with distinguished neighbourhood(*) well suited for matching verification. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 14
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 15
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 16
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 17
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 18
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 19
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 20
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 21
Corner Detection: Introduction + - >0 Demo of a point + with well distinguished neighbourhood. 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 22
Corner Detection: Introduction Example of detected points 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 23
Corner Detection: undistinguished patches: Basic principle distinguished patches: Image gradients ∇I(x, y) of undist. patches are (0, 0) or have only one principle component. Image gradients ∇I(x, y) of dist. patches have two principle components. ⇒ 9/10/2020 rank ( ∑ ∇I(x, y)* ∇I(x, y) ⊤ ) = 2 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 24
Algorithm (R. Harris, 1988) 1. filter the image by gaussian (2 x 1 D convolution), sigma_d 2. compute the intensity gradients ∇I(x, y), (2 x 1 D conv. ) 3. for each pixel and given neighbourhood, sigma_i: - compute auto-correlation matrix A = ∑ ∇I(x, y)* ∇I(x, y) ⊤ - and evaluate the response function R(A): R(A) >> 0 for rank(A)=2, R(A) → 0 for rank(A)<2 4. choose the best candidates (non-max suppression and thresholding) 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 25
Corner Detection: Algorithm (R. Harris, 1988) Harris response function R(A): R(A) = det (A) – k*trace 2(A) , [lamda 1, lambda 2] = eig(A) 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 26
Corner Detection: Algorithm (R. Harris, 1988) Algorithm properties: + “invariant” to 2 D image shift and rotation + invariant to shift in illumination + “invariant” to small view point changes + low numerical complexity - not invariant to larger scale changes - not completely invariant to high contrast changes - not invariant to bigger view point changes 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 27
Corner Detection: Algorithm (R. Harris, 1988) Exp. : Harris points and view point change 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 28
Corner Detection: Harris points versus sigma_d and sigma_i ↑ Sigma_d Sigma_I 9/10/2020 → (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 29
Corner Detection: Application 3 D camera motion tracking / 3 D reconstruction Algorithm: 1. Corner detection 2. Tentative correspondences - by comparing similarity of the corner neighb. in the searching window (e. g. cross-correlation) 3. Camera motion geometry estimation (e. g. by RANSAC) - finds the motion geometry and consistent correspondences 4. 3 D reconstruction - triangulation, bundle adjustment 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 30
Corner Detection: Camera Tracking Application - Boujou Input sequence 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 31
Corner Detection: Camera Tracking Application - Boujou Harris points 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 32
Corner Detection: Camera Tracking Application points consistent with 3 D camera motion 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 33
Corner Detection: Camera Tracking Application 3 D points and 3 D camera motion 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 34
Corner Detection: Camera Tracking Application 3 D animation 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 35
Corner Detection: Camera Tracking Application - Boujou Input sequence 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 36
Corner Detection: Camera Tracking Application - Boujou Harris points 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 37
Corner Detection: Camera Tracking Application - Boujou points consistent with 3 D camera motion 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 38
Acknowledgements for used demo images and software: • • • Jana Kostková, CMP - (slide 3: images, disparity map) Radim Šára, CMP - (slide 5: images, 3 D face reconstruction demo) Jan Kybic, CMP - (slide 10: medical image registration) • • • 2 d 3 – (slide 31 -38: Boujou Demo, slide 6: img. sequences) Imagineer. Systems – (slide 8: Mo. Key Demo, slide 8: img. sequences) The Pixel Farm – (slide 31 -33: img. sequence) 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 39
Synopsis http: //cmp. felk. cvut. cz/cmp/courses/dzo/resources/lecture_harris_urban. pdf 9/10/2020 (33 DZO, E 33 DZO) - Lecture 9 - Corner Detection and The Correspondence problem 40
- Slides: 40