A Personal Surround Environment Projective Display with Correction

A Personal Surround Environment: Projective Display with Correction for Display Surface Geometry and Extreme Lens Distortion Tyler Johnson, Florian Gyarfas, Rick Skarbez, Herman Towles and Henry Fuchs Department of Computer Science University of North Carolina at Chapel Hill IEEE VR– March 14, 2007

Multi-Projector Display 2 A Personal Surround Environment

Wide-FOV Projectors Conventional vs. fisheye lens projector Conventional Projector 3 Fisheye-Lens Projector A Personal Surround Environment

Outline Distortion due to Non-Planar Surfaces Lens Distortion Correction for Projectors Results Contributions and Future Work 4 A Personal Surround Environment

Non-planar Display Surfaces P 5 A Personal Surround Environment

Geometry Compensation via Ray Tracing Display Surface Projector Image Projector Viewer 6 A Personal Surround Environment

Classic Remapping Algorithm 2 -Pass Rendering (Raskar ’ 98) Pass 1: Pass 2: Render application scene Remap Pass-1 Image Projector Viewer 7 A Personal Surround Environment

2 -Pass Rendering P Display Surface Assumes no Geometry lens Pinhole distortion! Projection Pass-1 8 Pass-2 A Personal Surround Environment

Distortion with Wide-FOV Lenses Input Image 9 Projected Result A Personal Surround Environment

Previous Applications of Wide-FOV Projection Elumens Vision. Station Hemispherical display screen with fisheye-lens projector for immersive display. Static projector/display surface relationship. Distortion correction based on optical/mechanical design specifications. 10 A Personal Surround Environment

Previous Applications of Wide-FOV Projection Konieczny ’ 05, “A Handheld Flexible Display System” Allows exploration of volume data using a deformable hand -held screen. Dynamic display surface. 11 A Personal Surround Environment

Our Goals Our goals were to provide camera-based calibration for wide-FOV projectors geometric correction for arbitrary display surfaces support for tracked viewers 12 A Personal Surround Environment

3 -Pass Rendering Display Surface Geometry Pinhole Projection Pass-1 13 Lens Distortion Correction Pinhole Projection Pass-2 A Personal Surround Environment

3 -Pass Rendering Issues for Wide-FOV Lenses Fisheye lenses are highly non-linear Pinhole approximation in Pass-2 leads to sampling issues during lens distortion correction in Pass-3. 14 A Personal Surround Environment

Sampling Artifacts Display Surface P 3 -Pass Rendering All Passes at Projector Resolution (1024 x 768) 15 A Personal Surround Environment

Pass-3 Sampling Issues Fisheye and pinhole lenses have very different spatial sampling characteristics Leads to sampling issues during lens distortion correction. Over-sampled! Under-sampled! Over-sampled! Pinhole lens (input) Fisheye lens (output) 16 A Personal Surround Environment

Sampling Artifact Reduction Display Surface P Pass 1 -2: 4 x Projector Resolution (4096 x 3072) Pass 3: (1024 x 768) 17 A Personal Surround Environment

A New Approach Remove the pinhole approximation of the wide-FOV lens. Use the actual lens model of the projector. Allows simultaneous compensation for lens distortion and non-planar surface geometry. 18 A Personal Surround Environment

Changes to Correction Pipeline Projector Lens Model Display Surface Geometry Pinhole Projection Lens Distortion Correction Pinhole Projection Use actual projector lens model to avoid resampling artifacts! Pass-1 19 Pass-2 Pass-3 A Personal Surround Environment

A New Approach Pre-compute where projector rays intersect the display surface! Display Surface Projector Image Projector Viewer 20 A Personal Surround Environment

Implementation Pre-compute the projector to display surface (ray) mapping Store this projector-resolution, 2 D to 3 D map on GPU as a floating-point texture. Static map - independent of viewer location. New Pixel Shader At each projector pixel: • Look up <x, y, z> vertex on the display surface. • Project the vertex into ideal image to obtain output color. 21 A Personal Surround Environment

Results Display Surface P All Passes at Projector Resolution (1024 x 768) 22 A Personal Surround Environment

Calibration Capture structured light patterns with camera-pair Calibrate projector • Linear calibration followed by nonlinear optimization incorporating projector lens model Reconstruct surface geometry • RANSAC-based plane fitting technique [Quirk, EDT 06] • General reconstruction techniques can also be used 23 A Personal Surround Environment

Viewer Tracking Could apply simple 2 D-2 D mapping Applies only to stationary viewer! Our approach is independent of the viewer location Viewer location can be updated each frame 2 D - Mapping Only applicable for Stationary Viewer Pass-1 24 Combined Pass-2 & -3 Remapping A Personal Surround Environment

Heterogeneous 2 -Projector Display Also applicable to conventional projectors Display Surface Fisheye-lens Projector Conventional Projector 25 Hybrid Display with Conventional and Fisheye-lens Projectors A Personal Surround Environment

Results Video 26 A Personal Surround Environment

Contributions Single-pass geometric remapping algorithm Correcting for both extreme lens distortion and display surface geometry Minimizes sampling and FOV issues with fisheye lenses Simple GPU shader program implementation Static 2 D: 3 D mapping stored as 3 D floating point texture Demonstrated applicability Personal surround environment with a single fisheye lens projector Conventional or Wide-FOV fisheye AND Heterogeneous displays Applicable to real-time VR applications with user-head tracking 27 A Personal Surround Environment

Future Work Methods to further reduce sampling artifacts Mip-mapped Pass-1 textures for geometry optimized sampling. Application of more general lens models [Kannala, IEEE TPAMI, Aug. 06] Real-time, continuous calibration Accommodate projector or display surface changes while the application running. Dynamic vs. static Pass-2 re-map function. 28 A Personal Surround Environment

Thank You Funding support: ONR N 00014 -03 -1 -0589 DARWARS Training Superiority program VIRTE – Virtual Technologies and Environments program Elumenati LLC for loan of fisheye-lens projector 29 A Personal Surround Environment