Lowcost Photometric Calibration for Interactive Relighting Cline Loscos

Low-cost Photometric Calibration for Interactive Relighting Céline Loscos and George Drettakis l l 1 i. MAGIS*-GRAVIR/IMAG-INRIA Computer Sciences Department - University College of London

Context: augmented reality l Mix real and virtual worlds l Applications l l l Richer virtual experience l 2 entertainment (virtual studio) cinema (special effects) medical etc. real world references

Common illumination [Breen et al. 96] 3 Common illumination

Motivation l Applications in augmented reality l l l need of interactive systems simulation of realistic illumination needs time Find the right balance between l l l the capture process speed the response speed of the system the quality of the lighting simulation convincing results 4

Overview l State of the art l 5 relighting and remodelling for several known lighting conditions [Loscos et al. 99] l Photometric calibration l Conclusion

Realistic lighting simulation l Global illumination l l Lighting simulation l l 6 direct (from light sources) + indirect light (inter-reflections) Input - scene geometry + reflectance and emittance properties of surfaces Output - lit scene l Reflectance: describes the portion of light reflected l Classical methods: ray casting or radiosity

Inverse illumination l Goal l Inverse illumination [Sato et al. 97, Yu et al. 99, etc] l l 7 find radiometric properties (reflectance, light source exitance) real scene known independently of the original lighting conditions allows relighting input - lit scene output - reflectance estimation

Our relighting method [Loscos et al. 1999] l Interactive relighting of real scenes l Realistic common illumination l l Simple capture process l l 8 consistency of lighting between real and virtual few photos low-cost equipment

Assumptions 9 l Relighting from a single viewpoint l Diffuse scene l Direct lighting: ray tracing l Indirect lighting: hierarchical radiosity

Input data: reflectance estimate l Radiance images from a single viewpoint l a single light source per image Different lighting conditions 10

Reflectance estimate pixel per pixel l For each radiance image reflectance = radiosity / (direct light + indirect light ) Original photograph l 11 Estimated reflectance Indirect approximated by an ambient term

Merged reflectance confidence Merged reflectance x avg. x 12

Results of the method [Loscos et al. 1999] Video 13

Limitations in the reflectance estimate l Colours transformed by the camera l Inaccuracy of the reflectance estimate reflectance l loss of information: saturation, etc. 14 Reflectance pixels

Solution: High-Dynamic Range images l Radiance images [Debevec et al. 97] l l l 15 Input - several pictures from the same point of view at different shutter speeds - RGB values within integer range [0 -255] Output - camera’s response function - high-dynamic range of colours Remark: need to control the shutter speed

Adaptation: low-cost HDR images l New solution for a semi-automatic digital camera Kodac DC 260 No direct control of the shutter speed l Use of the EV parameter provided by the camera l 16

Adaptation: low-cost HDR images l 9 EV values [-2. . 2] = 9 different exposure times l EV = 0 : automatically chosen shutter speed l Use of the conversion typically used in photography l l to an arbitrary value ( EV = 0) Results in l 17 Fix better range of colours and less saturation

Limitations in the reflectance estimate l Problems l l Make radiance images consistent l l 18 several lighting conditions exposure time automatically selected by the camera inconsistent radiance values based on radiosity equation least squares solution

Make radiance images consistent l Algorithm l l 19 choose a reference radiance image compute a reference reflectance for the reference image (only for directly lit areas) compute an error factor for each radiance image apply this factor to get a consistent image

Limitations in the reflectance estimate l Incorrect illumination estimate l 20 incorrect estimate in shadow areas

Iterative algorithm for reflectance estimate l For each pixel: Initial reflectance (indirect = ambient) Indirect lighting iteration New reflectance l 21 convergence of reflectance values

Calibration results Reflectance RGB 22 Initial radiance After iterations

Calibration results reflectance Reflectance for a scanline (RGB) pixels 23

Calibration results reflectance Reflectance for a scanline (initial radiance) pixels 24

Calibration results reflectance Reflectance for a scanline (after iterations) pixels 25

Calibration results Reflectance (single exposure time) RGB 26 Initial radiance After iterations

Improvements due to calibration Reflectance (single exposure time) RGB 27 Initial radiance After iterations

Conclusion l Photometric calibration l l 28 improvement of the reflectance estimate quality respects the restrictions to the low-cost computation and equipment price

Future work l Improve the final display l l l Simplify the capture process l General perspectives l l 29 apply the response function of the camera apply a tone mapping specular effects moving viewpoint outdoor scenes toward real time