Vision Sensors for Stereo and Motion Joshua Gluckman

Vision Sensors for Stereo and Motion Joshua Gluckman Polytechnic University

Stereo Vision depth map

Stereo With Mirrors [ Gluckman and Nayar (CVPR 99)]

Why Use Mirrors? • Identical system response – Better stereo matching – Fastereo matching

Why Use Mirrors? • Identical system response – Better stereo matching – Fastereo matching • Data acquisition – No synchronization – Data Storage

Stereo Systems Using Mirrors Teoh and Zhang `84 Goshtasby and Gruver `93 Inaba `93 Mathieu and Devernay `95 Mitsumoto `92 Zhang and Tsui `98

Geometry and Calibration

Background – Relative Orientation p p` C C` R, t – 6 parameters

Background – Epipolar Geometry p C p` e e` C`

Background – Epipolar Geometry 3 p C p` e 4 e` Epipolar geometry – 7 parameters C`

Background – Epipolar Geometry 3 p C p` e 4 e` Epipolar geometry – 7 parameters C`

One Mirror – Relative Orientation mirror virtual camera

One Mirror – Relative Orientation virtual camera 3 parameters

One Mirror – Relative Orientation virtual camera 3 parameters

One Mirror – Epipolar Geometry 2 parameters – location of epipole

Two Mirrors – Relative Orientation D virtual camera

Two Mirrors – Relative Orientation virtual camera -1 = D D 1 D 2 = D 1 D 2 virtual camera D 2 -1 1 D camera

Two Mirrors – Relative Orientation virtual camera q 5 parameters camera

Two Mirrors – Epipolar Geometry 6 parameters 2 p V p` e 4 e` V`

Two Mirrors – Epipolar Geometry image of the axis m p epipole e p` epipole e`

Two Mirrors – Epipolar Geometry image of the axis m p epipole e p p` 1 1 p 2 p` 3 3 p 4 p` 4 epipole e`

Calibration Parameters Relative orientation Epipolar geometry Two Cameras 6 (rigid transform) 7 One Mirror 3 (reflection transform) 2 Two Mirrors 5 (screw transform) 6 Three+ Mirrors 6 (rigid transform) 7

Mirror Stereo Systems

Real Time Stereo System Calibrate Get Images Depth Map Rectify Matching

Rectification of Stereo Images Perspective transformations

Why Rectify Stereo Images? • Fast stereo matching • O(hw 2 s) O(hw 2) • Removes differences in rotation and scale

Not All Rectification Transforms Are the Same

Rectification – Previous Methods Ayache and Hansen `88 Faugeras `93 3 D methods – need calibration Robert et al. `93 Hartley `98 Loop and Zhang `99 2 D methods – rectify from epipolar geometry Roy et al. `97 Pollefeys et al. `99 Non-perspective transformations

The Bad Effects of Resampling the Images • Creation of new pixels causes – – • Loss of pixels – – [Gluckman and Nayar CVPR ’ 01] Blurs the texture Additional computation Loss of information Aliasing

Measuring the Effects of Resampling determinant of the Jacobian change in local area

Measuring the Effects of Resampling determinant of the Jacobian change in local area

Measuring the Effects of Resampling determinant of the Jacobian change in local area

Change In Aspect Ratio Preserves Local Area pixels created pixels lost

Skew Preserves Local Area aliasing

Minimizing the Effects of Resampling change in local area • P and P’ must be rectifying transformation • No change in aspect ratio and skew

The Class of Rectifying Transformations Fundamental matrix e e¢ Rotation and translation e e¢

The Class of Rectifying Transformations e e¢

The Class of Rectifying Transformations e e¢

The Class of Rectifying Transformations e e¢ e¢ e 6 parameters

The Class of Rectifying Transformations e e¢ no skew maintain aspect ratio e 2 parameters e¢

The Class of Rectifying Transformations scale perspective distortion 2 parameters

Finding the Best Rectifying Transform change in local area Find p 1 and p 8 that minimize e

Finding the Best Rectifying Transform change in local area Find p 1 and p 8 that minimize e • e is quadratic in p 1 so the optimal scale can be found as a function of p 8 • e is a 16 th degree rational polynomial in p 8

Finding the Best Rectifying Transform e 1 e 2 • e 1 and e 2 are symmetric convex polynomials • e 1 has a minimum at p 8 = 0 • e 2 has a minimum at p 8 = f 5 The minimum of e is between 0 and f 5

Finding the Best Rectifying Transform e 1 e 2 e 1 and e 2 depend on the location of epipoles at the same distance

Finding the Best Rectifying Transform e 1 e 2 e 1 and e 2 depend on the location of epipoles at a distance of 3 and 10

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize e

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize e Step 3: Construct P and P’

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize e Step 3: Construct P and P’ Step 4: Rectify the images using the perspective transformations

Rectification

Rectification and Stereo Matching

Rectified Stereo Using Mirrors Not rectified [Gluckman and Nayar CVPR ’ 00] Rectified

When Is a Stereo System Rectified? • • • No relative rotation between stereo cameras Direction of translation along the scan lines (x -axis) Identical intrinsic parameters (focal length)

Rectified Stereo Sensors 1 left virtual camera 4 3 5 2 right virtual camera D

Rectified Stereo Sensors 1 left virtual camera 4 3 5 2 right virtual camera

What Constraints Must Be Satisfied?

How Many Reflections? Even number of reflections Odd number of reflections

Example: Four mirrors Won’t Work

What Constraints Must Be Satisfied?

Single Mirror Rectified Stereo

Single Mirror Rectified Stereo b camera virtual camera

Three Mirror Rectified Stereo

Three Mirror Rectified Stereo 4 constraints n 1 , n 2 , n 3 and x-axis must be coplanar One constraint on the angles One constraint on the distances

A Three Mirror Solution 9 d. o. f. – 4 constraints = 5 parameter family of solutions

Sensor Size 9 d. o. f. – 4 constraints = 5 parameter family of solutions

Optimized Solutions

Rectified Stereo Sensors Mirrors

Rectified Images and Depth Maps

Misplacement of the Camera Mirrors

Misplacement of the Camera Mirrors Invariant to misplacement of camera center

Misplacement of the Camera Mirror Insensitive to tilt of optical axis Mirrors

Misplacement of the Camera Mirror Dependent on pan of optical axis Mirrors

Split Shot Stereo Camera Nikon Coolpix camera mirror attachment

Image Sensors for Motion Computation

Camera Motion motion rotation, translation, depth

éxù ê ú ê yú ê x¢ ú ê ú ë y¢û [ Anadan and Avidan (ECCV 00)] [e, e’]

[Gluckman and Nayar ICCV ’ 98] [Aloimonos et al]

Future Work