Interactive Virtual Relighting and Remodelling of Real Scenes

Interactive Virtual Relighting and Remodelling of Real Scenes C. Loscos 1, MC. Frasson 1, 2, G. Drettakis 1, B. Walter 1, X. Granier 1, P. Poulin 2 (1) i. MAGIS* - GRAVIR/IMAG - INRIA Rhône-Alpes * i. MAGIS is a joint project of CNRS/INRIA/UJF/INPG (2) Département d ’informatique et de recherche opérationnelle, Université de Montréal

Motivation Interior design Changes in lighting Geometric modification i MAGIS

Motivation ü Goal: interactive system l simple capture process l interactive ( ~ 1 sec. per frame) l modification of lighting l modification of geometry i MAGIS

Motivation ü We have to: l create a simple model of the real scene - geometry - approximate reflectance l represent real global illumination l develop interactive methods for modifications ü Goal is to be convincing, not highly accurate i MAGIS

Previous Work ü Geometric reconstruction l vision methods [Faugeras et al. 97, . . . ] (Realise) l constraint-based systems [Debevec et al. 96, Poulin et al. 98] l software: Photomodeler, etc. ü Reflectance recovery l e. g. , [Sato et al. 97, Ward 92, Debevec 98, Yu et al. 98, etc]. i MAGIS

Previous Work ü Real-time direct shadows l real point light source [State et al. 96] ü Common global illumination l non-interactive - [Nakamae et al. 86, Fournier et al. 93, Jancène et al. 95, Debevec 98, Yu et al. 99] l interactive - [Drettakis et al. 97, Loscos et al. 98] i MAGIS

Algorithm Overview ü Input ü Pre-process ü Interactive modification i MAGIS

Algorithm Overview - Assumptions ü Single viewpoint ü Diffuse assumption ü Lighting: radiosity = reflectance x ( direct light + indirect light ) l l direct lighting: ray casting indirect lighting: hierarchical radiosity i MAGIS

Simple Input Process ü Geometric reconstruction l several (4 -5) images from different viewpoints l geometric modelling using “Rekon” [Poulin et al. 98] ü Reflectance reconstruction l several (5 -7) images from a single viewpoint l different lighting conditions: single light source at different positions l “radiance images” i MAGIS

Input ü Radiance images from single viewpoint l combining multiple images reduces artefacts of estimation different lighting conditions i MAGIS

Pre-process ü Computation of approximate diffuse reflectance pixel by pixel l l compute individual reflectance images merge reflectance images using confidence values ü Initialise lighting system l data structure l hierarchical radiosity system i MAGIS

Reflectance Computation ü For each radiance image reflectance = radiosity / ( direct light + indirect light ) photograph reflectance i MAGIS

Confidence Images ü Estimate confidence l confidence ~ quality of reflectance estimate l create a confidence image per light source position ü Begin with confidence = Visibility l low in shadow regions ü Filtering process to remove unwanted effects l low for outliers (specular effects, light tripod) i MAGIS

Merged Reflectance Computation reflectance confidence merged reflectance x avg. x i MAGIS

Interactive Modification: Shadow Reprojection ü Direct illumination: pixel by pixel ü Indirect illumination: optimised radiosity solution Reflectance pixel Indirect lighting Direct lighting i MAGIS

Shadow Re-projection photograph simulated i MAGIS

Add/move/remove object (virtual or real) ü Visible surface changes: pixel by pixel local update l project bounding box of dynamic object - localise directly affected pixels original object insertion i MAGIS

Add/move/remove object (virtual or real) ü Direct lighting updates: shaft structure l localisation of visibility changes (shadows) l accelerate visibility computation (blocker lists) original object insertion i MAGIS

Add/move/remove object (virtual or real) ü Indirect illumination computed by a radiosity solution (optimised by the shaft structure) ü Example: moving object Position 1 Position 2 i MAGIS

Real Object Removal i MAGIS

Removing Real Objects ü Use of the reflectance image (lighting effects removed) to generate new textures reflectance images i MAGIS

Light Source Modification ü Insertion of a virtual light source l computation for every pixel - new form-factors - new visibility ü Indirect illumination: radiosity solution i MAGIS

Lighting Modification Original virtual lighting Insertion of virtual light i MAGIS

Video i MAGIS

Conclusion ü Input l data simple to acquire ü Pre-process l data structures optimised for fast updates ü Interactive modification l add and move virtual objects l remove real objects l relighting i MAGIS

Future Work ü Improve reflectance computation l use of high dynamic range images (instead of RGB) l better control of indirect illumination ü Allow motion of real objects ü Faster: parallel computation i MAGIS

Future Work ü Remove restrictions l diffuse reflectance [Yu et al. 99] l fixed view-point i MAGIS