Phase Correlation Bahadir K Gunturk 1 Phase Correlation

Phase Correlation Bahadir K. Gunturk 1

Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse function gives the translation amount between the images Bahadir K. Gunturk 2

Phase Correlation Bahadir K. Gunturk 3

Computer Vision Stereo Vision Bahadir K. Gunturk

Coordinate Systems n Let O be the origin of a 3 D coordinate system spanned by the unit vectors i, j, and k orthogonal to each other. i P O j Bahadir K. Gunturk k Coordinate vector 5

Homogeneous Coordinates n H P O Homogeneous coordinates Bahadir K. Gunturk 6

Coordinate System Changes n Translation Bahadir K. Gunturk 7

Coordinate System Changes n Rotation where Exercise: Write the rotation matrix for a 2 D coordinate system. Bahadir K. Gunturk 8

Coordinate System Changes n Rotation + Translation Bahadir K. Gunturk 9

Perspective Projection n Perspective projection equations Bahadir K. Gunturk 10

Review: Pinhole Camera Bahadir K. Gunturk 11

Review: Perspective Projection Bahadir K. Gunturk 12

Multi-View Geometry Relates • 3 D World Points • Camera Centers • Camera Orientations • Camera Parameters • Image Points Bahadir K. Gunturk 13

Stereo scene point p p’ image plane optical center Bahadir K. Gunturk 14

Finding Correspondences p Bahadir K. Gunturk p’ 15

Three Questions n n n Correspondence geometry: Given an image point p in the first view, how does this constrain the position of the corresponding point p’ in the second? Camera geometry (motion): Given a set of corresponding image points {pi ↔ p’i}, i=1, …, n, what are the cameras C and C’ for the two views? Or what is the geometric transformation between the views? Scene geometry (structure): Given corresponding image points pi ↔ p’i and cameras C, C’, what is the position of the point X in space? Bahadir K. Gunturk 16

Stereo Constraints M Image plane Y 1 Epipolar Line p p’ Y 2 Z 1 O 1 X 1 Focal plane Bahadir K. Gunturk X 2 O 2 Z 2 Epipole 17

Epipolar Constraint Bahadir K. Gunturk 18

From Geometry to Algebra P p p’ O’ O All vectors shown lie on the same plane. Bahadir K. Gunturk 19

From Geometry to Algebra P p O Bahadir K. Gunturk p’ O’ 20

Matrix form of cross product a=axi+ayj+azk b=bxi+byj+bzk Bahadir K. Gunturk a×b=|a||b|sin(η)u 21

The Essential Matrix Essential matrix Bahadir K. Gunturk 22

Stereo Vision n Two cameras. Known camera positions. Recover depth. Bahadir K. Gunturk 23

Recovering Depth Information P P’ 1 Q Q’ 1 P’ 2=Q’ 2 O 1 Depth can be recovered with two images and triangulation. Bahadir K. Gunturk 24

A Simple Stereo System LEFT CAMERA RIGHT CAMERA baseline Right image: target Left image: reference disparity Depth Z Elevation Zw Zw=0 Bahadir K. Gunturk 25

Stereo View Right View Left View Bahadir K. Gunturk Disparity 26

Stereo Disparity n The separation between two matching objects is called the stereo disparity. Bahadir K. Gunturk 27

Parallel Cameras P Z f xl xr pl pr Ol Or T Disparity: T is the stereo baseline Bahadir K. Gunturk 28

Finding Correspondences Bahadir K. Gunturk 31

LEFT IMAGE Correlation Approach (x , y ) l l For Each point (xl, yl) in the left image, define a window Bahadir K. Gunturk centered at the point n 32

RIGHT IMAGE Correlation Approach (x , y ) l l … search its corresponding point within a search region in Bahadir K. Gunturk 33 the right image n

RIGHT IMAGE (xr, yr) Correlation Approach dx (x , y ) l l … the disparity (dx, dy) is the displacement when the Bahadir K. Gunturk correlation is maximum n 34

Stereo correspondence n Epipolar Constraint q Reduces correspondence problem to 1 D search along epipolar lines epipolar line Bahadir K. Gunturk epipolar plane epipolar line 35

Stereo correspondence For each epipolar line For each pixel in the left image • Compare with every pixel on same epipolar line in right image • Pick pixel with the minimum matching error Of course, matching single pixels won’t work; so, we match regions around pixels. Bahadir K. Gunturk 36

? = Comparing Windows f g Most popular For each window, match to closest window on epipolar line in other image. Bahadir K. Gunturk 37

Comparing Windows Minimize Maximize Bahadir K. Gunturk Sum of Squared Differences Cross correlation 38

Feature-based correspondence n Features most commonly used: q Corners n Similarity measured in terms of: q q q surrounding gray values (SSD, Cross-correlation) location Edges, Lines n Similarity measured in terms of: q q Bahadir K. Gunturk orientation contrast coordinates of edge or line’s midpoint length of line 39

LEFT IMAGE corner Feature-based Approach line structure n For each feature in the left image… Bahadir K. Gunturk 40

RIGHT IMAGE corner Feature-based Approach line structure Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is Bahadir K. Gunturk maximum n 41

Correspondence Difficulties n Why is the correspondence problem difficult? q Some points in each image will have no corresponding points in the other image. (1) the cameras might have different fields of view. (2) due to occlusion. n A stereo system must be able to determine the image parts that should not be matched. Bahadir K. Gunturk 42

Structured Light n Structured lighting q q q Feature-based methods are not applicable when the objects have smooth surfaces (i. e. , sparse disparity maps make surface reconstruction difficult). Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth. Finding and matching such points is simplified by knowing the geometry of the projected patterns. Bahadir K. Gunturk 43

Stereo results q Data from University of Tsukuba Scene Bahadir K. Gunturk Ground truth (Seitz) 44

Results with window correlation Estimated depth of field (a fixed-size window) Bahadir K. Gunturk Ground truth (Seitz) 45

Results with better method A state of the art method Ground truth Boykov et al. , Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. Bahadir K. Gunturk (Seitz) 46

Window size W=3 n Effect of window size W = 20 Better results with adaptive window • • T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment, , Proc. International Conference on Robotics and Automation, 1991. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): 155 -174, July 1998 (Seitz) Bahadir K. Gunturk 47

Other constraints n n It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t change too quickly. ) Bahadir K. Gunturk 48

Parameters of a Stereo System n q q n P Intrinsic Parameters Characterize the transformation from camera to pixel coordinate systems of each camera Focal length, image center, aspect ratio Extrinsic parameters q q Xl Describe the relative position and orientation of the two cameras Rotation matrix R and translation vector T Bahadir K. Gunturk Pl p Yl Pr p l Zl Yr r Zr fl fr Ol Or R, T Xr 49

Applications First-down line courtesy of Sportvision Bahadir K. Gunturk 50

Applications Virtual advertising courtesy of Princeton Video Image Bahadir K. Gunturk 51