Shadingaware Multiview Stereo Fabian Langguth Kalyan Sunkavalli Sunil

Shading-aware Multi-view Stereo Fabian Langguth, Kalyan Sunkavalli, Sunil Hadap, Michael Goesele ECCV 2016 Presented by：Xiaohui Zhou April 25 th 2017 1

Multi-view Reconstruction input imagery posed imagery textured 3 D geometry reconstructed 3 D geometry 2

Relate work • Multi-view stereo • Combined Multi-view and Photometric Cues • Shading-based Refinement for General Illumination 3

Multi-view stereo • Operate on patches • Surface regularization to deal with structureless areas that are not well matched by photo consistency Problem: Furukawa et al 2010 Can not recover fine-scale surface details accurately 4

Combined Multi-view and Photometric Cues Nehab et al. photograph just use positions from multi-view stereo augmented with normal-map from photometric stereo Problem: combines position and normal Rely on a controlled and complex capture setup 5

Shading-based Refinement for General Illumination Wu et al. MVS result Improved result a shding model Problem: Require additional information: • A single , constant albedo • … Yu et al. 6

Summary Paper’s Approach t 1 poin y e K t 2 poin y e K 7

Land’s Retinex theory • Introduced by Land Mc. Cann in 1971 Input image Reflectance image Shading image 8

Land’s Retinex theory • Introduced by Land Mc. Cann in 1971 t n e i d a r g rong St Input image Reflectance image Shading image Affected by albedo 9

Land’s Retinex theory • Introduced by Land Mc. Cann in 1971 g n o r t S Input image nt e i d a r g Reflectance image Affected by albedo t n e i ad r g l l a m S Shading image Affected by lighting 10

Land’s Retinex theory • Two implications ◦ In multi-view reconstruction ，the geometric stereo term is usually accurate and robust in regions with strong gradients but fails for small gradients ◦ In regions of small gradients, we can factor the surface albedo out completely, resulting in an albedo-free shading term 11

Energy Formulation • Based on Retinex theory Energy Formulation： E = Eg + Geometric Error Es Shading Error 12

Geometric Error • 13

Geometric Error • In addition to constraints beween the main view and its neighbors, we also define pairwise terms between two neighbors rs o b gh i e n 6 g n i us This essentially measures the difference in error between neighbors and avoids overfitting to only one neighbor. 14

Lighting model • Lambertian reflectance • Third-order spherical harmonics basis functions Bh to approximate the incoming illumination 15

Shading error • Based on image gradients • Assumption : ◦ Measure the difference between the observed image gradient, ∇ I, and the gradient of the reflected intensity , ∇R --predicted by the lighting model • Simple Lambertian shading S • low-frequency spherical harmonics lighting l 16

Shading error • Based on image gradients • Assumption : ◦ Measure the difference between the observed image gradient, ∇ I, and the gradient of the reflected intensity , ∇R --predicted by the lighting model ? 17

Shading error • Based on image gradients • Assumption : ◦ Measure the difference between the observed image gradient, ∇ I, and the gradient of the reflected intensity , ∇R --predicted by the lighting model ? Solution： Optimize in logarithmic space 18

Shading error • Based on image gradients • Assumption : ◦ Measure the difference between the observed image gradient, ∇ I, and the gradient of the reflected intensity , ∇R --predicted by the lighting model 19

Shading Error • Retinex Theory ◦ Small gradients are caused solely by lighting ◦ Albedo gradient vanishes 20

Shading error • Based on image gradients • Assumption : ◦ Measure the difference between the observed image gradient, ∇ I, and the gradient of the reflected intensity , ∇R --predicted by the lighting model 21

Combined Energy • pixels with strong gradients the geometric stereo term • pixels with small gradients shading error ? w Ho a continuous trade-off between geometric stereo term and shading error 22

Combined Energy Strong gradient Small gradient Visualization of the trade-off for an input image. Based on the magnitude of the image gradient ，for every pixel we use mainly stereo term (dark regions) or our shading term (bright regions). 23

Combined Energy 24

Surface Representation and Optimization • A framework of Semerjian to optimize this energy function[Semerjian, B. ECCV 2014] Contribution： ◦ smoothness：Bicubic patches ◦ surface representation with a continuous definition of depth values and surface normals 25

Surface Representation 26

Surface Representation 27

Final Optimization • Surface is represented as set of bicubic patches in coarse scales • Patches interpolate depth values and surface normals smoothly over a set of pixels • Coarse to fine optimization by splitting and adding/removing patches • Smaller patch sizes of 8× 8 and lower are then optimized using new energy 28

Results • use a variety of datasets • use 6 -9 neighbor images 29

Results Galliani, S Furukawa, Y 30

Results 16 images 2 neighbors Galliani, S Furukawa, Y 31

Results 32

Results Zollhoefer et al. 2015 33

Results 34

Conclusion • Key ideas： ◦ A combined multi-view stereo and shading-based reconstruction that balances the two terms 35

Conclusion • Key ideas： ◦ A combined multi-view stereo and shading-based reconstruction that balances the two terms ◦ Gradient-based trade-off between stereo and shading energies 36

Conclusion • Key ideas： ◦ A combined multi-view stereo and shading-based reconstruction that balances the two terms ◦ Gradient-based trade-off between stereo and shading energies ◦ Albedo-free：treats spatially varying albedo implicitly 37

Future work • Problem： ◦ Limited by the basic lighting model ◦ Disable: • Self-shadowing • Indirect illumination • Specular materials • Future work ◦ To create a more robust system that can be applied to more complex scenes 38

39

Question From Weizhi • 40

Question From Weizhi • Can you explain the vanishes of albedo gradient ���� (x)？ ◦ Small gradients are caused solely by lighting, if gradient of intensity I is small，the albedo is local constant, then its gradient is close to 0. That is Albedo gradient vanishes. This indicates an albedo invariance which can also be thought of in the following way: If the albedo is locally constant, an intensity gradient is only caused by a change in surface normals, and given a lighting model, the surface normals have to change in a particular direction which does not depend on the actual value of the albedo. 41

Question From Yunpeng • 42

Question From Jingyi • How to use energy function to affect the surface reconstruction？ ◦ ◦ ◦ Given the surface representation using bicubic patches. Every patch is defined by 4 nodes, and A node itself represents 4 optimization variables: the depth, the first derivatives of the depth and the mixed second derivative. For every patch , the paper use a Gauss-Newton type solver to optimize the nonlinear energy function using the four variables. If the patch defined by nodes which have high errors, this patch will be removed. Coarse to fine optimization by splitting and adding/removing patches until get the finest scale that patches cover a 2× 2 set of pixels 43

Question From Jianhong • Can you make a introduction to the Bicubic Patches？ ◦ ◦ Bicubic patches is the most common use of Bézier surfaces. A single patch has 16 control points, one at each corner, and the rest positioned to divide the patch into smaller sections. Bicubic patches are useful surface representations because they allow an easy definition of surfaces using only a few control points. It has many advantages, such as the following: • Easy to manipulate • Have much better continuity properties. In this paper, the key reason that it use this bicubic patches to represent the surface is bicubic patch can provide a continuous definition of depth values and surface normals. 44

Question From Guohang • 45