A Distributed Cooperative Framework for Continuous Multi Projector

A Distributed Cooperative Framework for Continuous Multi. Projector Pose Estimation Tyler Johnson, Greg Welch, Henry Fuchs, Eric La Force, and Herman Towles Department of Computer Science University of North Carolina at Chapel Hill IEEE VR 2009 - March 16, 2009

Funding ONR: Behavior Analysis and Synthesis for Intelligent Training (BASE-IT), Dr. Roy Stripling, Program Manager ONR: Virtual Technologies and Environments Program (VIRTE), CDR Dylan Schmorrow, Program Manager ONR-NAVAIR: Deployable Intelligent Projection Systems for Training, SBIR contract with Renaissance Sciences Corporation IARPA: Mockup Future Analyst Workspace (A-Desk), Dr. Jeff Morrison, Program Manager NSF: Integrated Projector-Camera Modules for the Capture and Creation of Wide-Area Immersive Experiences, CRI: IAD grant 2 A Distributed Cooperative Framework for Continuous Calibration

Adaptive Multi-Projector Displays An Intelligent Projector Unit (IPU) 3 A Distributed Cooperative Framework for Continuous Calibration

Challenges Geometric Compensating for display surface shape Co-registration of projection images No Compensation Photometric Intensity blending in image overlaps Matching colors between projectors 4 A Distributed Cooperative Framework for Continuous Calibration

Geometric Calibration Before Display Use During Display Use Render Imagery Geometric Image Correction Project Structured Light Concurrently Calibrate Projectors Estimate Display Surface 5 Continuous Calibration Calibrate Projectors Estimate Display Surface A Distributed Cooperative Framework for Continuous Calibration

Continuous Calibration A Calibrated Two-Projector Display Projectors Moved or Bumped A Recalibrated Two-Projector Display 6 A Distributed Cooperative Framework for Continuous Calibration

Cooperative Calibration We propose a distributed, Kalman filter-based approach to continuous calibration where intelligent projector units interact as peers to cooperatively estimate the poses (orientation & position) of all projectors during actual display use 7 A Distributed Cooperative Framework for Continuous Calibration

Related Work Continuous Calibration Active (Calibration Patterns) • Imperceptible Structured Light [Cotting 04, 05] Passive (Application Imagery) • Continuous Display Surface Estimation [Yang&Welch 01] • Single Projector Pose Estimation[Johnson&Fuchs 07] • Multi-Projector Pose Estimation [Zhou 08] Hybrid • Automatic switch from passive to active [Zollmann 06] Distributed Upfront Calibration [Bhasker 06] 8 A Distributed Cooperative Framework for Continuous Calibration

Contributions Our Kalman filter-based distributed cooperative framework provides Continuous pose estimation for multiple projectors • Compatible with both active and passive feature collection • All projectors may move simultaneously Temporal filtering Fault tolerance & scalability 9 A Distributed Cooperative Framework for Continuous Calibration

Cooperative Calibration Peer-to-Peer based, Single Program, Multiple Data Each IPU considers itself to be the “local” IPU Other IPUs are considered “remote” IPUs Each IPU is responsible for calculating its own pose using local and remote correspondences Assumptions The internal calibration of each IPU is fixed and known The geometry of the display surface is static and known Projectors remain mostly stationary, however they may drift over time or occasionally be moved by the user 10 A Distributed Cooperative Framework for Continuous Calibration

Local Correspondences Measured for each IPU between its primary and secondary cameras Provides an estimate of pose 11 A Distributed Cooperative Framework for Continuous Calibration

Remote Correspondences Measured between the primary camera of the local IPU and a remote IPU Remote IPU acts as a reference in estimating pose of local IPU 12 A Distributed Cooperative Framework for Continuous Calibration

Collection of Measurements Display Surface 13 A Distributed Cooperative Framework for Continuous Calibration

Kalman Filter Measurement Function Measurements in Estimate , using Measurements in Pose of Intrinsics of Display Surface Model Pose of Display Surface Model 14 Intrinsics of Predicted Measurements in A Distributed Cooperative Framework for Continuous Calibration

Kalman Filter Pose of State Pose of 15 Error Covariance A Distributed Cooperative Framework for Continuous Calibration

Filter Update Predict IPUs remain stationary Time Update (t) Add additional uncertainty Correct state based on residual Measurement Update Measurement Jacobian 16 Measurement Noise A Distributed Cooperative Framework for Continuous Calibration

Distributed Operation Each IPU has local access to Its own intrinsic calibration & pose Its own camera images Display surface model Kalman filter update requires remote access to Intrinsic calibration & poses of remote IPUs Error & process noise covariance of remote IPUs Images from primary cameras of remote IPUs captured at time 17 A Distributed Cooperative Framework for Continuous Calibration

Distributed Operation Request/response mechanism for exchanging camera images, pose information etc 18 A Distributed Cooperative Framework for Continuous Calibration

Results Before Movement 1240 x 1160 -95 During Movement 2. 85 After Movement Ψ 2. 7 y -0. 3 θ rad mm -120 2640 2560 19 -0. 7 z 0. 35 φ 0. 15 A Distributed Cooperative Framework for Continuous Calibration

Video Distributed Cooperative Pose Estimation in a Two-IPU display P 20 P A Distributed Cooperative Framework for Continuous Calibration

Discussion Observability & Drift P P Surface geometry may not fully constrain pose Possible to “anchor” solution in unobservable directions Cooperative Estimation Not required for a working system Ensures imagery is registered between projectors, especially when pose may be unobservable 21 A Distributed Cooperative Framework for Continuous Calibration

Future Work Continuous calibration of display surface Dynamic projector refocusing Dynamic photometric blending Improve scalability 22 A Distributed Cooperative Framework for Continuous Calibration

Future Applications 23 A Distributed Cooperative Framework for Continuous Calibration

In Conclusion 24 A Distributed Cooperative Framework for Continuous Calibration