Online Tracking of Outdoor Lighting Variations for Augmented

Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras Yanli Liu 1, 2 and Xavier Granier 2, 3, 4 1: College of Computer Science, Sichuan University, P. R. China 2: INRIA Bordeaux Sud-Ouest, France 3: LP 2 N (CNRS, Univ. Bordeaux, Institut d'Optique) 4: La. BRI (CNRS, University of Bordeaux)

Motivation o Augmented reality n mobility VR 20 12

Motivation o Two consistency n Geometric consistency o Devices o Camera position n n GPS, UWB, Omnisense Wi. Fi, cell information o Camera pose n Linear accelerometers o Tracking via computer vision o [Cornelis et al. LNCS 2001, zhang et al. CVPR 2007, Xu et al. image and Vision Computing 2008] n Illumination consistency o outdoor lighting is largely dependent on weather and time VR 20 12

Motivation o Two problems n Online process o first step toward real-time solutions n Moving viewpoints o Handhold camera jitter VR 20 12

Previous Work o Markers or lighting probes [Debevec Siggraph’ 98, Agusanto ISMAR’ 03, Kanbara ICPR’ 04, Hensley I 3 D’ 07] n too dense sampling n our method does not require any supplemental devices Debevec Siggraph’ 98 VR 20 12

Previous Work o Three components of shading n BRDF n geometry n lighting original image rendered image [Wang PG’ 02] o Fix other one or two components Li ICCV’ 03, Hara PAMI’ 05, Andersen ICPR’ 06, Sun ICCV’ 09] [Wang PG’ 02, n 3 D reconstruction n controlled environment (indoor or lab) VR 20 12

Previous Work o Time-lapse outdoor video analysis Siggraph’ 07, Sunkavalli CVPR 08] n take whole video sequence as input n Post-processing VR 20 12 [Sunkavalli Siggraph’ 07] [Sunkavalli

Previous Work o Learning based outdoor illumination estimation [Liu TVC’ 09, Liu CAVW’ 10, Xing C&G’ 11] n offline stage learning n fixed viewpoint VR 20 12 Liu CAVW’ 10 moving viewpoints

Our Method o Key ideas n Tracking illumination variation by tracking feature points n Feature points tracking is error prone. n Select reliable feature points using global illumination constraint and spatial-temporal coherency. VR 20 12

Illumination and BRDF model o Outdoor lighting [Sunkavalli SIG’ 07, Sunkavalli CVPR 08, Madsen In. Tech 2010] n the sunlight o directional light o colored intensity o sun direction n the skylight o ambient light o colored intensity VR 20 12

Illumination and BRDF model o Neutral reflection model [Lee PAMI’ 90, Montoliu LNCS’ 05, Eibenberger ICIP 2010, ICCV 2011] n the color of the specular reflection is the same as the color of the incident lighting. o Phong-like model VR 20 12

System Initialization n Tracking illumination variation by tracking plane segmentation feature points [Hoiem IJCV’ 07] o 3 D geometry vs normals o planar feature points KLT feature-points mean-shift color segmentation first frame VR 20 12 threshold-based Shadow detection clustered feature-points

System Initialization o BRDF initialization n pixels difference at in sun lit regions depend on specular parameters and : n Assuming piecewise constant n Spatially varying diffuse VR 20 12 , and

Tracking Lighting Variation with Reliable Feature Points o Energy function Alignment-based weight n Outdoor lighting is nearly constant during time intervals less than 1/5 second. VR 20 12 control the smooth degree of skylight

Tracking Reliable Features and Their Attributes o Feature points labeling n Three attributes: o Normal (plane, homography matrix) o BRDF parameters o Shadow situation VR 20 12 Spatial & temporal coherency

Tracking Reliable Features and Their Attributes o Feature points labeling current point is not in shadow paired point is labeled in compute lighting t -1 t VR 20 12

Results and Discussion o Quantitative results n PC: Intel i 7 2. 67 GHz and 6 GB RAM n MATLAB n Video resolution 640 480 Average fps and average number of feature points estimated on 1, 000 frames VR 20 12

Results and discussion o Quantitative results Average percentage of different steps in total computational cost VR 20 12

Results and Discussion o Visual results n Building scene n Wall scene VR 20 12

Conclusion o Fully image-based pipeline n online tracking of lighting variations of outdoor videos. o Manages lighting changes and misalignment of feature points o Ensure a stable estimation on a sparse set feature points. VR 20 12

Limitations and Future Work o Rough shadow detection n 3 D reconstruction vs shadow detection n Sun-lit features o Initialization n automatic initialization: easy but may fail in some cases n manual initialization: may be tedious for a non-expert user. n Semi-assisted initialization VR 20 12

Limitations and Future Work o Tracking independently on R, G, and B channels n priori model of outdoor illumination color or spectra n difficult to optimization o The first step of a long march to a seamless and real-time AR solution for videos with moving viewpoints. VR 20 12

Thanks for your attention! Questions? VR 20 12