Structure from Motion A moving cameracomputer computes the

Structure from Motion A moving camera/computer computes the 3 D structure of the scene and its own motion CSE 803 Fall 2008 Stockman 1

Sensing 3 D scene structure via a moving camera We now have two views over time/space compared to stereo which has multiple views at the same time. CSE 803 Fall 2008 Stockman 2

Assumptions for now n n n The scene is rigid. The scene may move or the camera may move giving a sequence of 2 or more 2 D images Corresponding 2 D image points (Pi, Pj) are available across the images CSE 803 Fall 2008 Stockman 3

What can be computed n n The 3 D coordinates of the scene points The motion of the camera Camera sees many frames of 2 D points Rigid scene with many 3 D interest points From Jabara, Azarbayejani, Pentland CSE 803 Fall 2008 Stockman 4

From 2 D point correspondences, compute 3 D points and TR CSE 803 Fall 2008 Stockman 5

applications n n We can compute a 3 D model of a landmark from a video We can compute the trajectory of the sensor relative to the 3 D object points CSE 803 Fall 2008 Stockman 6

Use only 2 D correspondences, Sf. M can compute 3 D jig pts … up to one scale factor. CSE 803 Fall 2008 Stockman 7

http: //www 1. cs. columbia. edu/~je bara/htmlpapers/SFM/sfm. html Jabara, Azarbayejani, Pentland a) Two video frames with corresponding 2 D interest points. 3 D points can be computed from Sf. M method. b) Some edges detected from 2 D gradients. c) Texture mapping from 2 D frames onto 3 D polyhedral model. d) 3 D model can be viewed arbitrarily! CSE 803 Fall 2008 Stockman 8

Virtual museums n n Much work, and software, from about 10 years ago. 3 D models, including shape and texture can be made of famous places (Notre Dame, Taj Mahal, Titanic, etc. ) and made available to those who cannot travel to see the real landmark. Theoretically, only quality video is required. Usually, some handwork is needed. CSE 803 Fall 2008 Stockman 9

Sf. M methods n n n Typically require careful mathematics EX: from 5 matched points, get 10 equations to estimate 10 unknowns; also a more popular 8 pt linear method Methods must consider effects of noise See Faugeras et al and **** Methods can run in real time CSE 803 Fall 2008 Stockman 10

Special mathematics n n n Epipolar geometry Fundamental matrix: computed from a pair of cameras Essential matrix: specialization of fundamental matrix when calibration is available CSE 803 Fall 2008 Stockman 11

Finishing the course: 2 options n n HW 7 on stereo computation (Sat 6 Dec) (60%/7) Final Exam (25%): Monday 8 Dec 12: 45 -2: 45 n n Sf. M report (6 Dec) and demonstration. Find existing code and get it to work on CSE 803 images (60%/7 + 10%) Final Exam (15%) limited problems CSE 803 Fall 2008 Stockman 12