3 D Stereo Reconstruction using i Phone Devices

  • Slides: 26
Download presentation
3 D Stereo Reconstruction using i. Phone Devices Final Presentation 24/12/2012 Performed By: •

3 D Stereo Reconstruction using i. Phone Devices Final Presentation 24/12/2012 Performed By: • Ron Slossberg • Omer Shaked 1 Supervised By: • Aaron Wetzler

Project’s Goal • Building a self-contained mobile 3 D reconstruction system using two i.

Project’s Goal • Building a self-contained mobile 3 D reconstruction system using two i. Phone devices 2

Background – Pinhole Camera Model • The basic camera model • Transformation from 3

Background – Pinhole Camera Model • The basic camera model • Transformation from 3 D to 2 D coordinates: • Distortion also taken into account 3

Background – Stereo Vision • Combine images from two cameras to generate depth image

Background – Stereo Vision • Combine images from two cameras to generate depth image • Relative cameras’ positions in physical space (R, T) and image planes space (F) retrieved by stereo calibration process 4

Background – Stereo Correspondence • Open. CV offers a number of algorithms for stereo

Background – Stereo Correspondence • Open. CV offers a number of algorithms for stereo correspondence • We chose to use two algorithms that offer a good compromise between efficiency and quality: – Block Matching – Semi Global Block Matching 5

Background – Stereo Correspondence (Cont. ) • Block Matching: – Looks at blocks of

Background – Stereo Correspondence (Cont. ) • Block Matching: – Looks at blocks of pixels along the epipolar lines and finds matches according to cross correlation. – Example of typical result: 6

Background – Stereo Correspondence (Cont. ) • Semi Global Block Matching: – Adds on

Background – Stereo Correspondence (Cont. ) • Semi Global Block Matching: – Adds on to the normal block matching algorithm by introducing global consistency constraints – The constraints are introduced by aggregating matching costs along several independent, onedimensional paths across the image – Example of typical result: 7

Background – Reconstructed Scene • The matching algorithm produces a disparity map which is

Background – Reconstructed Scene • The matching algorithm produces a disparity map which is a gray scale image where every color corresponds to a certain disparity and thus a certain depth • Using a reprojection matrix on the disparity map we obtain a point in space corresponding to each pixel • We can render these points as a 3 d mesh using the original picture colors for each vertex. 8

Programming Environment • i. Phone app programming – Objective-C programming – Model-View-Controller design pattern

Programming Environment • i. Phone app programming – Objective-C programming – Model-View-Controller design pattern • Main implemented features: – Displaying and controlling the views – Inter-device communication and time synchronization – Persistent storage of data 9

Programming Environment (cont. ) • Open. CV libraries – C++ open-source code – Implement

Programming Environment (cont. ) • Open. CV libraries – C++ open-source code – Implement all the required algorithms for performing the calibration and reconstruction processes – Handle the interface with the i. Phone’s camera • Main implemented features: – Integrating open. CV functions into our i. Phone app – Correct data flow into and out of every open. CV function 10

Programming Environment (cont. ) • Open. GL ES libraries – A lightweight version of

Programming Environment (cont. ) • Open. GL ES libraries – A lightweight version of the open-source Open. GL libraries, which includes an i. OS API – Implement the framework for rendering 2 D and 3 D computer graphics • Main implemented features: – Displaying the reconstructed images as an interactive 3 D surface 11

Software High-Level Design Main Menu Connect Devices • Generating a Bluetooth session between the

Software High-Level Design Main Menu Connect Devices • Generating a Bluetooth session between the devices 12

Software High-Level Design Main Menu Settings • Setting the right parameters for stereo calibration

Software High-Level Design Main Menu Settings • Setting the right parameters for stereo calibration and reconstruction 13

Software High-Level Design Main Menu Calibration • Performing stereo calibration for the devices using

Software High-Level Design Main Menu Calibration • Performing stereo calibration for the devices using a chessboard pattern 14

Software High-Level Design Main Menu Reconstruction • Performing 3 D stereo reconstruction of the

Software High-Level Design Main Menu Reconstruction • Performing 3 D stereo reconstruction of the images captured by the devices 15

Software High-Level Design Main Menu Photo Album 3 D Image Display • Interactive 3

Software High-Level Design Main Menu Photo Album 3 D Image Display • Interactive 3 D color display of the images • Disparity map images shown within table 16

TIME FOR LIVE DEMO AND MOVIE 17

TIME FOR LIVE DEMO AND MOVIE 17

Calibration Process Flow Initial State Calibration Bluetooth session created Message delay calculated Validating message

Calibration Process Flow Initial State Calibration Bluetooth session created Message delay calculated Validating message delay Initializing camera Initial State Calibration Capture Wait Message Delay Send Capture Indication Send Capture Order Capture Image 18 Extract Corners Exchange image corners data Extract Corners

Calibration Process Flow (Cont. ) Performed separately at each device Calibrate Process was separated

Calibration Process Flow (Cont. ) Performed separately at each device Calibrate Process was separated to increase accuracy Compute Intrinsic Parameters Compute Extrinsic Stereo Parameters Save Parameters 19

Reconstruction Process Flow Initial State Reconstruction 20 Bluetooth Session Created Message delay calculated Calibration

Reconstruction Process Flow Initial State Reconstruction 20 Bluetooth Session Created Message delay calculated Calibration Performed Validating message delay Initializing camera Initial State Reconstruction Load parameters Compute undistorted rectified bitmap Choose Reconstruction Algorithm

Save Disparity Image Reconstruction Process Flow (Cont. ) Capture Wait Message Delay Send Capture

Save Disparity Image Reconstruction Process Flow (Cont. ) Capture Wait Message Delay Send Capture Indication Send Capture Order Capture Image Send and Receive Image 21 Send Images Send and Receive Image Remap Images to get Rectified Images Compute Disparity by stereo correspondence Save Disparity Image

Implementation Issues • Simultaneous Photo Capture – Problem: need devices to capture images at

Implementation Issues • Simultaneous Photo Capture – Problem: need devices to capture images at the same time to gain good results – Solution: implemented RTT calculation algorithm that ignores anomalous results and performs update phases during operation – Other Solutions: Web Service, GPS – Advantage: messages traverse only short distance, no dependency on GPS signals – Main Disadvantage: GPS achieves better accuracy 22

Implementation Issues (Cont. ) • Inter-Device Communication – Problem: need to pass messages and

Implementation Issues (Cont. ) • Inter-Device Communication – Problem: need to pass messages and data between the two devices – Solution: Bluetooth communication – Other Solutions: Wi-Fi – Advantages: existing easy-to-use framework, simple protocol with low overhead – Main Disadvantage: smaller BW (affects the duration of the reconstruction process) 23

Summary • Very challenging and enjoyable project • Introduction with both computer vision and

Summary • Very challenging and enjoyable project • Introduction with both computer vision and mobile app development • Final outcome is a stable, user-friendly app providing live results • Helpful documentation for future usage – Detailed webpage and demo movie • Finally – many thanks to Aaron for his guidance and support throughout this project ! 24

References • Computer Vision course, Spring 2010, University of Illinois – http: //www. cs.

References • Computer Vision course, Spring 2010, University of Illinois – http: //www. cs. illinois. edu/~dhoiem/courses/vision_spring 10/lectures/ • Developing Apps for i. OS, Paul Hegarty, Stanford fall 2010 course (available on i. Tunes) • Multiple View Geometry in Computer Vision course, University of North Carolina – http: //www. cs. unc. edu/~marc/mvg/slides. html • Modeling the Pinhole camera, course lecture, University of Central Florida • Computer Vision tutorial, GIP lab • Learning Open. CV, Gary Bradski & Adrian Kaehler • Stereo Vision using the Open. CV library by Sebastian Droppelmann, Moos Hueting, Sander Latour and Martijn van der Veen 25

The End COMMENTS & QUESTIONS 26

The End COMMENTS & QUESTIONS 26