Image Processing Seminar Shadow Detection and Removal Using

Image Processing Seminar Shadow Detection and Removal Using Video Advisor: Dr. Hagit Hel-Or By: Ido Yerushalmy February 2008

Shadow Using JPEG From Characteristics Video JPEG Format Incentives Shadow Quant. Table Characteristics Why remove a shadow? • The problem: shadows, like real objects, usually differ from their surroundings and move with the object creating them – making it difficult to distinguish between the 2 types • Removing the shadows can help perform segmentation, pattern matching and moving objects extraction • Knowing the luminance of the scene allows adding CG sections in a realistic manner MPEG Survey Format GOP ML correlation Algorithm MLle Doub enhancement Quantization Summary

Shadow From Video Incentives Shadow Characteristics Survey ML Algorithm ML enhancement Summary Types of shadows A cast shadow is the area projected on the scene by an object. It can be divided into: 1. Umbra – direct light is totally blocked 2. Penumbra – light partially blocked

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics MPEG Format Survey ML GOP Algorithm correlation MLle Doub enhancement Quantization Modeling appearance of a point 1. For simplicity, assume all surfaces are matt (Lambertian world) 2. Each point we see is the result of its reflectance and the amount of luminance it receives In a mathematical notation: I(x, y) = R(x, y) * L(x, y) Summary Image = Reflectance x Luminance

Shadow Using JPEG From Characteristics Video Incentives Shadow Characteristics Modeling appearance of a point– Cont. Quant. Table MPEG Format Survey We can furthermore model the umbra and penumbra cases: ML GOP Algorithm correlation MLle Doub enhancement Quantization Illuminate Summary Penumbra Umbra Where: • CA – ambient light intensity • CP – light source intensity • L – direction of light source • N(x, y) – object surface normal • K(x, y) – softening due to penumbra

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics MPEG Format Survey ML GOP Algorithm correlation MLle Doub enhancement Quantization Summary Survey of common methods Algorithms assume the following: 1. Light source is white, spread evenly and strong 2. The reference image is assumed static, textured and planar

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics MPEG Format Survey ML GOP Algorithm correlation MLle Doub enhancement Quantization Summary Statistical non-parametric algorithms 1. Such statistical algorithms assume that shadows have similar chromaticity but lower brightness than the background model. 2. A statistical learning procedure is used to automatically determine the appropriate thresholds.

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics MPEG Format Survey ML GOP Algorithm correlation MLle Doub enhancement Quantization Summary Deterministic non-model based with color exploitation 1. Uses the HSV color representation

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics MPEG Format Survey ML GOP Algorithm correlation MLle Doub enhancement Quantization Summary Deterministic non-model based with color exploitation – cont. 2. A shadow cast does not change the Hue significantly 3. Shadows often lower the saturation and value of the points

Shadow Using JPEG From Characteristics Video Deterministic non-model based with color exploitation – cont. Incentives Shadow Quant. Table Characteristics MPEG Format Survey Formally, it can be described as: ML GOP Algorithm correlation MLle Doub enhancement Quantization Summary Where: • I and B are the input image and its computed background • Alpha defines the “power” of the shadow and Beta is used only to avoid misclassifications due to noise

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics MPEG Format Survey ML GOP Algorithm correlation MLle Doub enhancement Quantization Summary Deterministic non-model based with spatial redundancy exploitation Three criteria are tested to detect shadows: 1. The presence of a "darker" uniform regions hints a cast shadow 2. Presence of high difference in luminance with relation to the reference frame 3. The presence of static and moving edges. Static edges hint static shadows. Looking at the width of the edges can reveal if this is a penumbra shadow, since it usually have a softer (= wider) edge Drawback: This method is not enough generic and has too many assumptions

Shadow Using JPEG From Characteristics Video Incentives Maximum Likelihood Algorithm Shadow Quant. Table Characteristics Assumptions on the world: Survey MPEG Format 1. Lambertian (all surfaces are matt) MLGOP Algorithm correlation MLle Doub enhancement Quantization Summary 2. A single reflectance image (static camera), and many illuminations images. Fits a surveillance camera model Formally denoted by: For simplicity reasons we will work in the log domain:

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Survey MPEG Format Maximum Likelihood Algorithm - cont Statistical knowledge from previous works: When derivative filters are applied to natural images, the filter output tends to be sparse MLGOP Algorithm correlation MLle Doub enhancement Quantization Summary Natural Image Horizontal Derivative Hist.

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Maximum Likelihood Algorithm - cont What is this histogram similar to? Survey MPEG Format MLGOP Algorithm correlation MLle Doub enhancement Quantization Summary Horizontal Derivative Hist. Laplace Given an image, lets try to find its reflectance part by fitting the derivative’s histogram to the expected Laplace. This can be done by taking the pixel wise median over several frames (Maximum Likelihood algorithm)

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Maximum Likelihood Algorithm - cont In a formal manner: Survey MPEG Format MLGOP Algorithm correlation MLle Doub enhancement Quantization Summary Where: • On – is the nth derivative filter • The median is for each pixel over a series of frames Note: this is true as long as the reflectance image is indeed constant Retrieving the luminance image is now easy: l(x, y, t) = i(x, y, t)-r(x, y)

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Survey MPEG Format MLGOP Algorithm correlation MLle Doub enhancement Quantization Summary Maximum Likelihood Algorithm - cont Retrieving the reflectance (and luminance) images:

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Maximum Likelihood Algorithm - cont Another example: Survey MPEG Format MLGOP Algorithm correlation MLle Doub enhancement Quantization Summary Frame 2 Frame 11 ML reflectance ML illumination 2 How do you produce such a sequence of images? ML illumination 11

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Survey MPEG Format MLGOP Algorithm correlation MLle Doub enhancement Quantization Summary Maximum Likelihood Algorithm - cont A common way is to use a device like this:

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Survey MPEG Format ML GOP Algorithm correlation le MLDoub Enhancement Quantization Summary ML enhancement • This algorithm is based on the one we just explored • Uses a static camera view (same as before) • Does not assume a Lambertian world, resulting in a changing reflectance image.

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Survey MPEG Format ML GOP Algorithm correlation le MLDoub Enhancement Quantization Summary ML enhancement For example: The difference in reflectance at the border between white stripes and a black road is not constant, since the white stripes change their reflectance as a factor of time (light direction). This causes such areas to be considered (falsely) as part of the illumination image.

Shadow Using JPEG From Characteristics Video Incentives ML enhancement – cont. Shadow Quant. Table Characteristics Note that the problematic areas are where textures exist Survey MPEG Format Overcoming the problem: 1. Start by generating a basic (ML) reflectance image 2. Try to move the textures from the illumination image to the reflectance image. ML GOP Algorithm correlation le MLDoub Enhancement Quantization Summary One way to do it is:

Shadow Using JPEG From Characteristics Video Incentives Shadow Quant. Table Characteristics Survey MPEG Format ML GOP Algorithm correlation le MLDoub Enhancement Quantization Summary ML enhancement – cont. This (almost) solves the problem we saw before:

Shadow Using JPEG From Characteristics Video Incentives JPEG Format Shadow Quant. Table Characteristics Survey ML Algorithm ML enhancement Summary 1. Shadow removal can be helpful as a pre-processing stage in segmentation and pattern matching algorithms 2. Shadow modeling is difficult. Some assumptions must be made. 3. Some of the assumptions introduced: Lambertian world, natural image statistics, constant reflectance image … 4. Assuming a changing reflectance gives better result, but relies on additional assumptions

Using JPEG Characteristics JPEG Format Quant. Table References 1. A. Prati, R. Cucchiara, I. Mikic, M. M. Trivedi, "Analysis and detection of shadows in video streams: a comparative evaluation" in Proceedings of Computer Vision and Pattern Recognition conference (CVPR 2001), vol. 2, pp. 571 -576, 2001 MPEG Format GOP correlation Double Quantization 2. Y. Weiss. “Deriving intrinsic images from image sequences”. In ICCV 01, pages II: 68 -75. IEEE, 2001 Summary 3. Y. Matsushita and K. Nishino, “Illumination normalization with time-dependent intrinsic images for video surveillance, ” IEEE Trans. Pattern Anal. Mach. Intell. , 26, pp. 1336 -1348 (2004) 4. Athinodoros S. Georghiades Peter N. Belhumeur, “From Few to Many: Illumination Cone Models for Face Recognition Under Variable Lighting and Pose”, IEEE Trans. on Pattern Analysis and Machine Intelligence