Image Compositing and Matting Introduction l l l
- Slides: 50
Image Compositing and Matting
Introduction l l l Matting and compositing are important operations in the production of special effects. These techniques enable directors to embed actors in a world that exists only in imagination, or to revive creatures that have been extinct for millions of years. During matting, foreground elements are extracted from a film or video sequence. During compositing, the extracted foreground elements are placed over novel background images.
Traditional approaches to matting l l l Traditional approaches to matting include blue-screen matting and rotoscoping. The former requires filming in front of an expensive blue screen under carefully controlled lighting The latter demands talent and intensive user interaction.
Rotoscoping l Rotoscoping --- the process of tracking contours in a video sequence
Basic problem of blue screen matting l l Given an image of a foreground object shot in front of a backing color (blue screen or green screen or other colors) Obtain a matte of the foreground object so that the foreground object can be blended into a new background image using the matte to produce a new composite image.
Notations l l l l Let C = [R, G, B] denote a color with 0 ≤ R, G, B ≤ 1. Let a denotes a transparency value with 0 ≤ a ≤ 1. Foreground image color: Cf = [Rf , Gf , Bf ], af = 1 Backing (screen) color: Ck = [0, 0, Bk], ak = 1 (assuming blue screen) Original foreground object color: Co = [Ro, Go, Bo] Background image color: Cb = [Rb, Gb, Bb], ab = 1 Composite image color: Cc = [Rc, Gc, Bc]
Problem Statement l l Given Cf and Cb at corresponding pixels, and Ck a known backing color, and assuming Cf = ao. Co + (1 − ao)Ck Determine ao and Co, which then gives the composite color Cc = ao. Co + (1 − ao)Cb at the corresponding point, for all points that Cf and Cb share in common.
Solution l Since Cf = ao. Co + (1 − ao)Ck , we have: R f = a o. R o G f = a o. G o Bf = ao. Bo + (1 − ao)Bk. l 4 unknowns, 3 equations…
Case 1: No Blue l l There is no blue in Co, i. e. , Bo=0, and Bk≠ 0. Then This case is very restrictive. It rules out many colors, including grays because grays have blue.
Case 2: Gray and Skin Color l l Assume Ro=a. Bo (or Go=a. Bo) and Bk≠ 0 Then where
Example 1: Gray l Then l This case applies to science fiction movie in which the spaceships are mostly gray.
Example 2: Skin Color l This case applies to human faces, hands, legs, etc.
Solution for General Cases l To solve the matting problem in general: l l Need to take the same image with two different backing colors. This gives four equations for solving the four unknowns Case 1: Use Two Different Shades of Blue. Case 2: Use Two Different Backing Colors.
Case 1 l Use two different shades of blue Bk 1 and Bk 2 as backing colors. l Then l so
Case 2 l Use two different backing colors Ck 1 and Ck 2. l Or l Over-constrained, 1 unknowns, three equations
Case 2: Solution 1 l Adding up 3 equations:
Case 2: Solution 2 l Apply least squares method Define: l Then l Least-squares l
Least-Squares l Set
Case 1 Example
Case 2 Example
Other Developments in Matting l l Matting Without Blue Screen A method proposed by Ruzon and Tomasi l l l User specify object region and boundary region. Alpha value of object region is set to 1. Alpha value of boundary region is computed by estimating the contributions of neighboring objects’ colors.
Example
Bayesian Approach
Shadow Matting l l l Pull a matte of shadow. Acquire photometric and geometric properties of the target scene by sweeping oriented linear shadows across it. Then, composite the shadow onto the scene.
Example
Environmental Matting Left: alpha matte. Middle: environment matte. Right: photo.
References 1. 2. 3. 4. 5. Y. -Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski. A bayesian approach to digital matting. In Proc. IEEE CVPR, pages II– 264–II– 271, 2001. Y. -Y. Chuang, D. B. Goldman, B. Curless, D. H. Salesin, and R. Szeliski. Shadow matting and compositing. ACM Transactions on Graphics, 22(3): 494– 500, July 2003. M. A. Ruzon and C. Tomasi. Alpha estimation in natural images. In Proc. IEEE CVPR, pages 18– 25, 2000. A. R. Smith and J. F. Blinn. Blue screen matting. In Proc. ACM SIGGRAPH, pages 259– 268, 1996. D. E. Zongker, D. M. Werner, B. Curless, and D. H. Salesin. Environment matting and compositing. In Proc. SIGGRAPH, pages 205– 214, 1999.
Digital Compositing l l l Digital compositing means “digitally manipulated integration of at least two source images to produce a new image. ” The new image must appear realistic. It must be completely and seamlessly integrated, as if it were actually photographed by a single camera.
Example 1
Example 2
More examples l http: //www. beezlebugbit. com/digital/efx_t op. htm
Main Topics l l Alpha blending: blending foreground and background Keying: separating foreground and background l l Luma, chroma, difference keying Rig removal: removing unwanted elements
Alpha Blending l l l C = [α F + (1 – α) B] If α = 1, then C = F, foreground is shown, i. e. , foreground is opaque. If α = 0, then C = B, background is shown, i. e. , foreground is transparent. 0 < α < 1: semi-transparent, e. g. , shadow, smoke, etc. If α ranges from 0 to 255, then the formula becomes: C = [α F + (1 – α) B] / 255
Example: No Background
Example: With Background
Note l For shadow, a must take fractional value (0 < α < 1). Otherwise, shadow looks unreal.
Boundary area l a at boundary area should also be fractional. Otherwise, have dark fringes; unrealistic.
Summary l A good matte has fractional a in shadow, and along object boundaries and shadow boundaries.
Keying l l l Separating foreground from background, creating a matte of foreground. Also called pulling a matte (of foreground), or keying out (i. e. , making transparent) background. Recall: l A good matte has fractional a in shadow, and along object boundaries and shadow boundaries.
Basic methods l l l Luma keying: based on luminance (i. e. , intensity) Chroma keying: based on color (i. e. , blue screen, green screen) Difference keying: requires a clean plate, i. e. , a background image without the foreground element.
Basic Idea l l Compute difference between foreground and background (based on luma, chroma, or color) Very small diff a = 0. Very large diff a = 1. Intermediate diff intermediate a
Luma Keying l l Key out the background based on luminance. Useful when background has a uniform luminance that is very different from foreground luminance.
Result
Chroma Keying l l l Key out the background based on color. Useful when background has a uniform color that is very different from foreground color. Example: Image shot with blue screen.
Characteristics of blue screen image
Difference Keying l l More general than luma and chroma keying. Key out background based on pixel-wise color difference between foreground and background footage.
Final Composition
Rig Removals l l l Rigs are equipment that support the actors or the props. Sometimes, rigs cannot be removed by keying alone. So, have to apply masking technique to remove rigs. Need clean plate of background footage. If camera moves, then need motion-controlled camera: Computer controls camera to move the same way twice: l l Without foreground objects; get clean plate. With foreground objects.
Basic Idea l l l Apply a mask to mask out the rig. Then, replace pixels in masked area by corresponding pixels in clean plate background. If rig moves in footage, then have to animate the mask accordingly.
- Exposure sheet
- Automatic portrait segmentation for image stylization
- Blue screen matting
- Spectral matting
- Fba matting
- Flebectomia
- Vray render elements compositing photoshop
- Compositing digital images
- Digital compositing
- Rendering realtime compositing
- Vray back to beauty
- Digital compositing
- Rendering realtime compositing
- Digital compositing
- Analog image and digital image
- Real vs virtual image
- Virtual vs real image
- Image transform in digital image processing
- Noise
- Compression in digital image processing
- Image segmentation in digital image processing
- Subjective fidelity criteria in digital image processing
- Image sharpening in digital image processing
- Static image vs dynamic
- Geometric transformation in digital image processing
- Gravitational blood drop
- Zooming and shrinking in digital image processing
- Ce n'est pas une image juste c'est juste une image
- Difference between logical file and physical file
- Qual o resultado de: image image
- Enlighten about image noise and restoration
- Image transforms in digital image processing
- Image geometry in digital image processing
- Noise
- "image.so.com" -site:image.so.com
- Generative adversarial networks
- Introduction to digital image processing
- Introduction to digital image processing with matlab
- Essay structure introduction
- Power-law (gamma) transformation
- Module 5 supply and demand introduction and demand
- Tybalt act 1 scene 1
- Fourier transform formula
- A neural image caption generator
- Spiral progression in science
- Image analysis and exploitation
- Representation and description in digital image processing
- Indexed color image
- For coordinates p(2,3) the 4 neighbors of pixel p are
- Hamid rahim sheikh
- Image alignment