3 D computer vision techniques KH Wong 3

3 D computer vision techniques KH Wong 3 D computer vision techniques v. 4 b 2 1

Seminar Title: 3 D computer vision techniques. l Abstract In this talk, the ideas of obtaining 3 D information of objects (or called 3 D reconstruction) using different techniques are discussed. Currently, the most popular one is the image based method that uses 2 D cameras for 3 D reconstruction; in particular reconstruction based on one-image, two-image and multipleimage are discussed. Moreover, batch and sequential treatments of input data are studied. I will also talk about novel techniques, such as using multiple cameras and laser based methods to obtain 3 D information. And I will discuss how 3 D computer vision is used in film and game production. Finally naked-eye 3 D display technologies will be mentioned. 3 D computer vision techniques v. 4 b 2 2

Overview (part 1) l l Introduction From 2 D to 3 D l l l Camera systems/calibration Feature extraction/correspondence Reconstruction algorithms l l l Previous projects l l l Virtual viewer/ Projector camera systems Keystone correction Novel setups l l 2 views, 3 views , N views Real-time algorithms/Kalman filter Multiple cameras/ Camera array Obtain 3 D directly l l Structured light Laser approach Kinect approach Photometric stereo 3 D computer vision techniques v. 4 b 2 3

Overview (part 2) l Applications l l l Photos from tourists (photo tourism) http: //phototour. cs. washington. edu/ 3 D displays Possible future research l l l Classification based on 3 D information Content search 3 D based on 3 D keys Merging with sound information 3 D computer vision techniques v. 4 b 2 4

Motivation l l l We live in a 3 D world We see 2 D images but perceive the world in 3 D Intelligent robot should have this 3 D reconstruction capability 3 D computer vision techniques v. 4 b 2 5

How to obtain 3 D information? l l Cameras-2 D Range sensors-3 D 3 D computer vision techniques v. 4 b 2 6

Challenges l Obtain 3 D information for tasks in a 3 D world. l l l 2 D-to-3 D reconstruction from a camera 3 D directly— laser range sensor, kinect sensor Novel sensors l l l Camera array/ multiple camera One pixel camera light field camera 3 D computer vision techniques v. 4 b 2 7

2 D-to-3 D reconstruction (feature based method) l l l Camera (perspective projection) Features-extraction and correspondences Methods l l One-image method Two-image (Stereo) method Three-image method N-image method l l Bundle adjustment Kalman filter 3 D computer vision techniques v. 4 b 2 8

Camera: 3 D to 2 D projection Screen or CCD sensor Y World center 9 Perspective model u=F*X/Z (nonlinear relation) v=F*Y/Z Virtual v Z F Thin lens or a pin hole F Real Screen Or CCD sensor

Perspective Projective l Yw Zw World Coordinates Rc, Tc Xw Model M at t=1 v-axis X, Y, Z image Xc-axis Zc-axis Principal axis u-axis F=focal length (u, v) Camera Coordinates. Oc = c (Image center, (0, 0, 0) ox, oy) (Camera center) Yc-axis 3 D computer vision techniques v. 4 b 2 (0, 0) of image plane 10

In paintings l l l Western Fresco by Raphael, 1510 1511, Stanza della Signatura, Vatican Palace, Rome. Chinese 《富春山居圖》是 元朝畫家黃公望的 作品，創作於 1347年至 1350年 Dwelling in the Fuchun Mountains (富春 山居圖) by Huang Gongwang (1269 – 1354) 3 D computer vision techniques v. 4 b 2 11 http: //www. es. flinders. edu. au/~mattom/science+society/lectures/illustrations/lecture 17/schoolathens. html http: //jsl 641124. blog. 163. com/blog/static/17702514320115219508530/

Feature correspondences --Camera moved, find correspondences for neighboring images --We can use feature to identify the motions of projected 3 D features in 2 D. l Area a Image at t=t 0 (or left image) 3 D computer vision techniques v. 4 b 2 Image at t=t 0+dt (or right image) 12

Demo l l Youtube Movie http: //www. youtube. com/watch? v=azl. DGK 6 e 1 U 3 D computer vision techniques v. 4 b 2 13

One-image 2 D-to-3 D reconstruction 3 D computer vision techniques v. 4 b 2 14

One image 2 D-to-3 D reconstruction method l Difficult and with ambiguity 3 D computer vision techniques v. 4 b 2 http: //ai. stanford. edu/~asaxena/reconstruction 3 d/ 15

One image 2 D-to-3 D l Using prior knowledge (e. g. face) http: //www. wisdom. weizmann. ac. il/~ronen/papers/ Hassner Basri - Example Based 3 D Reconstruction from Single 2 D Images. pdf 3 D computer vision techniques v. 4 b 2 16

Two-image 2 D-to-3 D reconstruction 3 D computer vision techniques v. 4 b 2 17

Two-image 2 D-to-3 D reconstruction method: stereo vision l Objectives: l l Basic idea of stereo vision Stereo reconstruction by epipolar geometry l l Stereo camera pair calibration (find Fundamental matrix F) Construct the 3 D (graphic) model from 2 images Inside a computer 3 D computer vision techniques v. 4 b 2 Graphic model 18

if camera motion is pure translation : Triangular calculation l Left Camera Principle axis Object Right Camera Px(x, y, z) Principle axis By similar triangle, w. r. t left camera lens center z Left Image plane X’l X’r Right Image plane Focal Length f b (Baseline) Left camera center Horizontal (reference point) Disparity=x. L-x. R By similar triangle, w. r. t right camera lens center 3 D computer vision techniques v. 4 b 2 One major problem is to locate x’l and x’r The correspondence problem 19

If camera motion is NOT pure translation : Use Epipolar Geometry X l l. Right_image_point. T*E*left_image_point=0 Left side is the reference Left epipolar line 1 Focal length=f 1 (x 2, y 2) Right epipolar line (x 1, y 1) left Frame Plane-1 1 O Plane-3 3 Perpendicular to T X 2 or T X 1 e 1 right Frame Plane-2 2 R, T Base line=||T|| 3 D computer vision techniques v. 4 b 2 e 2 O 2 Focal length=f 2 20

Method: 8 -point algorithm http: //www. cs. manchester. ac. uk/ugt/COMP 37111/papers/Hartley. pdf l Find 8 point corresponded ( l l Map 8 Right_image_points to left_image_point Solve the epeiolar formula l l Right_image_point. T*E*left_image_point=0 Find E. From E we can find camera R (rotation) , T (translation) From R, T we can find model (3 D positions of the left feature points (using left as reference) 3 D computer vision techniques v. 4 b 2 21

An example of stereo reconstruction l l An example Short-Baseline Stereo Systems for Mobile Devices l http: //www. lelaps. de/videos. html#SQx 5 v. U 8 BA-M http: //www. lelaps. de/projects/stmobile. html 3 D computer vision techniques v. 4 b 2 22

Stereo-based Free-space Estimation l Another example http: //www. lelaps. de/videos. html#Vr. KBNt. AN 03 o lhttp: //www. lelaps. de/projects/freespace. html 3 D computer vision techniques v. 4 b 2 23

Three-image 2 D-to-3 D reconstruction 3 D computer vision techniques v. 4 b 2 24

Three-image 2 D-to-3 D reconstruction method l l More robust using 3 views It contains 3 epipolar relations l l l Stereo 1: view 1, 2 , Stereo 2: view 2, 3, Stereo 3 : view 3, 1. Combine 3 epipolar geometry information together. Similar to the algorithm in epipolar geometry (apply 3 times) http: //www. cs. unc. edu/~marc/tutorial/node 45. html 3 D computer vision techniques v. 4 b 2 M=3 -D model point M, m’, m” are image points C, C’, C” are camera centers 25

Example of 3 -image reconstruction l Example LIBVISO: Feature Matching for Visual Odometry http: //www. youtube. com/watch? v=DPLh 6 Mox. PAk 3 D computer vision techniques v. 4 b 2 26

N-image 2 D-to-3 D reconstruction (batched method: order of images can be random ) 3 D computer vision techniques v. 4 b 2 27

N-image 2 D-to-3 D reconstruction method l Bundle adjustment approach l l Guess iteratively the solution for 3 D to explain the measurements of feature points in all images Math: Q(u, v)=g(X), g is nonlinear (projection) because l u=f. X/Z l v=f. Y/Z, f=focal length l Given Q (image measurement) , we want to find X=(X, Y, Z)i from image points (u, v)i of all N model points (i=1, , , N), g is the projection formulas l A typical non linear optimization problem, l Gauss-Newton for non linear optimization method is used. 3 D computer vision techniques v. 4 b 2 28

Batched method: order of images can be random l From measurement [u, v]I find X X v 1 [u, v]1 v 2 [u, v]2 v 3 vm [u, v]3 O 1 Image R 1, T 1 t=1 … [u, v]m … O 2 R 2, T 2 Image t=3 3 D computer vision techniques v. 4 b 2 O 3 R 3, T 3 Image t=m Om Rm, Tm 29 Camera motion

Example l Bundle adjustment reconstruction http: //www. cse. cuhk. edu. hk/%7 Ekhwong/demo/canyon 2 b 2. mpg 3 D computer vision techniques v. 4 b 2 30

N-image 2 D-to-3 D reconstruction (Sequential method: order of images are used like in a move ) 3 D computer vision techniques v. 4 b 2 31

Sequential method: order of images are used like in a move l From measurement [u, v]I find X X v 1 [u, v]1 v 2 [u, v]2 v 3 vm [u, v]3 O 1 Image R 1, T 1 t=1 … [u, v]m … O 2 R 2, T 2 Image t=3 3 D computer vision techniques v. 4 b 2 O 3 R 3, T 3 Image t=m Om Rm, Tm 32 Camera motion

Kalman Filter Prediction pictures by Ko Hoi Fung 3 D computer vision techniques v. 4 b 2 Correction 33 33

Kalman filter example Prediction t’ = 1: Position = x 1’ Velocity = v 1’ x 1’ = v 0 * t + x 0 t = 0: Position = x 0 Velocity = v 0 States: • Position • Velocity Measurements: t = 1: Position = x 1 Update • Position 3 D computer vision techniques v. 4 b 2 34 34

Example l Hernan Badino and Takeo Kanade: "A Head-Wearable Short. Baseline Stereo System for the Simultaneous Estimation of Structure and Motion". IAPR Conference on Machine Vision Applications (MVA), Nara, Japan, June 2011 http: //www. youtube. com/watch? v=SQx 5 v. U 8 BA-M 3 D computer vision techniques v. 4 b 2 35

Novel sensors : Camera array/ Multiple camera systems l Camera array/ multiple camera: High Performance Imaging - Using Large Camera Array http: //www. youtube. com/watch? v=0 W_1 Ce 2 l. TBo http: //graphics. stanford. edu/papers/Camera. Array/ 3 D computer vision techniques v. 4 b 2 36

The Self-Reconfigurable Camera Array Each camera Demo movie http: //chenlab. ece. cornell. edu/projects/Mobile. Cam. Array/videos/train. mov http: //chenlab. ece. cornell. edu/projects/Mobile. Cam. Array/videos/self_reconfiguration. mov 3 D computer vision techniques v. 4 b 2 37 http: //chenlab. ece. cornell. edu/projects/Mobile. Cam. Array/

Applications 3 D computer vision techniques v. 4 b 2 38

Photo tourism l http: //phototour. cs. washington. edu/ 3 D computer vision techniques v. 4 b 2 39

Projector-camera system Application of computer vision 3 D computer vision techniques v. 4 b 2 40

A Projector-Camera system 3 D computer vision techniques v. 4 b 2 41

Projector-Camera calibration l http: //www. youtube. com/watch? v=YHh. QSglmuq. Y&feature=channel_page 3 D computer vision techniques v. 4 b 2 42

Our setup l 3 D computer vision techniques v. 4 b 2 43

Calibration procedure l 3 D computer vision techniques v. 4 b 2 44

Quadrangle tracking l 3 D computer vision techniques v. 4 b 2 45

Experiments l 3 D computer vision techniques v. 4 b 2 46

Projection result l 3 D computer vision techniques v. 4 b 2 47

Results l 3 D computer vision techniques v. 4 b 2 48

Hand held direct manipulation 3 D Display l http: //www. youtube. com/watch? v=v. VW 9 QXu. Kfo. Q&feature=relmfu 3 D computer vision techniques v. 4 b 2 49

Keystone correction l Configuration 3 D computer vision techniques v. 4 b 2 50

Aim of this work l Desired Results Keystoned projection Corrected projection 3 D computer vision techniques v. 4 b 2 51

Overview l Three major modules l l l Projector-camera pair calibration Projection region detection and tracking Automatic keystone correction Flow chart 3 D computer vision techniques v. 4 b 2 52

Pre-warp projection image Pre-warped projection image Display result http: //www. youtube. com/watch? v=y 5 XYdeh 8 Bno&list=UUfy 2 Eumi. HMeo. Uor. MFR 0 wo. ZA&index=1&feature=plcp 3 D computer vision techniques v. 4 b 2 53

Keystone correction l Some real correction results 3 D computer vision techniques v. 4 b 2 54

Obtain 3 D directly l Laser range sensor l l Time of flight Kinect 3 D computer vision techniques v. 4 b 2 55

Photometric stereo l Lamertian light formula http: //www. wisdom. weizmann. ac. il/~vision/photostereo/ • Given 3 or more known light source we can find the normal N • From the set of N we can approximate the surface http: //www. taurusstudio. net/research/photex/ps/equation. htm 3 D computer vision techniques v. 4 b 2 56

Photometric stereo using multiple cameras and multiple light sources l Demo Dynamic Shape Capture using Multi-View Photometric Stereo SIGGRAPH 2009 http: //www. youtube. com/watch? v=9 hgs 5 z. N 38 lk 3 D computer vision techniques v. 4 b 2 57

Multiple cameras fro human body reconstruction l 3 D computer vision techniques v. 4 b 2 Homepage: //media. au. tsinghua. edu. cn 58

Experimental Results 3 D computer vision techniques v. 4 b 2 59 3 D Modeling Using MVML Dome 59 2020/11/21

Multiple camera doom l http: //www. mpi-inf. mpg. de/~yliu/ 3 D computer vision techniques v. 4 b 2 60

Structured light method l Calculate the shape by how the strip is distorted. l http: //www. laserfocusworld. com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home. html 3 D computer vision techniques v. 4 b 2 61

Real time Virtual 3 D Scanner Structured Light Technology l Demo http: //www. youtube. com/watch? v=a 6 pgz. NUjh_s 3 D computer vision techniques v. 4 b 2 62

Time of flight laser method l l Send the IR-laser light to different directions and sense how each beam is delayed. Use the delay to calculate the distance of the object point http: //www. laserfocusworld. com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home. html 3 D computer vision techniques v. 4 b 2 63

LIDAR light detection and ranging scanner http: //www. youtube. com/watch? v=Muw. QTc 8 KK 44 l Leica terrestrial lidar (light detection and ranging) scanner 3 D computer vision techniques v. 4 b 2 http: //hodcivil. edublogs. org/2011/11/06/lidar-%E 2%80%93 -light-detection-and-ranging/ http: //commons. wikimedia. org/wiki/File: Lidar_P 1270901. jpg 64

3 D Laser Scanning Underground Mine Mapping l Demo http: //www. youtube. com/watch? v=BZbvz 8 fe. Pe. Q 3 D computer vision techniques v. 4 b 2 65

Motion capture for film production (MOCAP) l IR light emitter and camera http: //www. youtube. com/watch? v=Ix. Jrhnynl. N 8 lhttp: //upload. wikimedia. org/wikipedia/commons/7/73/Motion. Capture. jpg 3 D computer vision techniques v. 4 b 2 lhttp: //www. naturalpoint. com/optitrack/products/s 250 e/indepth. html 66

3 D body scanner l http: //www. youtube. com/watch? v=86 h. N 0 x 9 Ryc. M 3 D computer vision techniques v. 4 b 2 http: //www. cyberware. com/products/scanners/ps. html http: //www. cyberware. com/products/scanners/wbx. html 67

3 -D Face capture l http: //www. youtube. com/watch? v=-TTR 0 Jrocs. I&feature=related 3 D computer vision techniques v. 4 b 2 http: //www. captivemotion. com/products/ 68

Dimensional Imaging 4 D Video Face Capture with Textures l http: //www. youtube. com/watch? v=Xt. TN 7 t. Wa. XTM&feature=related Dimensional Imaging 4 D Video Face Capture with Textures 3 D computer vision techniques v. 4 b 2 69

Kinect l l Another structure light method Use dost rather than strips http: //www. laserfocusworld. com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home. html 3 D computer vision techniques v. 4 b 2 70

Kinect Hardware l 3 D computer vision techniques v. 4 b 2 71

See the IR-dots emitted by KINECT l http: //www. youtube. com/watch? v=-gbz. Xjd. Hf. JA http: //www. youtube. com/watch? v=d. TKl. NGSH 9 Po&feature=related 3 D computer vision techniques v. 4 b 2 72

Novel sensors : light field camera Spin off from Stanford camera array l l light field camera : LYTRO camera Be able to refocus after the picture is taken http: //www. youtube. com/watch? v=7 QV 152 jc 3 Ac 3 D computer vision techniques v. 4 b 2 https: //www. lytro. com/camera 73

light field camera How does it work l 3 D computer vision techniques v. 4 b 2 http: //www. quora. com/Lytro/How-does-the-new-Lytro-camera-work 74

3 D (Volumetric) display Rendering for an Interactive 360º Light Field Display SIGGRAPH 2007 Papers Proceedings l http: //www. youtube. com/watch? v=h 6 a. UIS 44 ez. E http: //gl. ict. usc. edu/Research/3 DDisplay/ 3 D computer vision techniques v. 4 b 2 75

Occlusion-Capable Volumetric 3 D Display by Cossairt, etal. Actuality Systems, Inc l http: //www. youtube. com/watch? v=8 Ka. Qmn 2 VTzs 3 D computer vision techniques v. 4 b 2 http: //www. 3 dcgi. com/cooltech/displays. htm 76

3 D display Using a lattice with thin slits, viewer's eyes see different pixels on the screen to create 3 d perception l http: //www. televisions. com/tv-articles/TV-in-3 D/Displaying-3 D-Without-Glasses. php 3 D computer vision techniques v. 4 b 2 77

The future l l Content search in 3 D video data bases Shot boundary detection Video data mining Video classification 3 D computer vision techniques v. 4 b 2 78

Appendix 3 D computer vision techniques v. 4 b 2 79

Essential matrix E (a 3 x 3 matrix) P. 110[2] X 1 is 3 -D X in left camera (reference) system X 2 is 3 -D X in right camera system Exercise 1: Draw vectors T X 2 or T X 1 in the diagram l 3 D computer vision techniques v. 4 b 2 80

Essential Matrix E l Right_image_point. T*E*left_image_point=0 3 D computer vision techniques v. 4 b 2 81