Color Image Processing in the block DCT Space

Color Image Processing in the block DCT Space Jayanta Mukhopadhyay Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur, 721302, India jay@cse. iitkgp. ernet. in 1

What is COLOR? Selective emission/reflectance of different wavelengths 2

Color Spectrum Illumination Reflectance • Spectrum: Intensity as a function of wavelength. 3

Color Stimuli Illumination Reflectance • The colour of an object: is the product of the spectrum of the incident light with the light absorption and/or reflection properties of the object. 4

What is Perceived Color? • The response generated by a stimulus in the cones gives the perceived color • Three responses 5

Human color perception • For human eye • – Approximately 65% of all cone are sensitive to red light • – 33% are sensitive to green light • – 2% are sensitive to blue light • But blue cones are the most sensitive. 6

Tri-stimulus Values • Integration over wavelength X = ∫C(λ)x(λ) dλ = Σ C(λ)x(λ) Y = ∫C(λ)y(λ) dλ = Σ C(λ)y(λ) Z = ∫C(λ)z(λ) dλ = Σ C(λ)z(λ) Real colors span a subset of the XYZ space. • Two different stimuli can have same XYZ values. –Metameris • Additive color mixtures modeled by addition in XYZ space. • 7

Amounts of three primaries needed to match all wavelengths of the spectrum The curves represented by the cone’s reception are not simple peaks. They are, instead, quite complex curves. They even go negative! RGB is not capable of reproducing every single color we can see. 8

Perceived Color Features • Intensity – Sum of the spectrum – Energy under the spectrum • Hue – Mean wavelength of the spectrum – What wavelength sensation is dominant? • Saturation – Standard deviation of the spectrum – How much achromatic/gray component? • Chrominance – Hue and saturation 9

Limitation of Tri-Stimulus Model • No physical feel as to how colors are arranged. • How do brightness change? • How does hue change? • Subtractive like paint cannot be modeled by XYZ space. 10

CIE XYZ Space • Intensity (I) – X+Y+Z • Chrominance (x, y) - (X/I, Y/I) –Chromaticity chart – Projection on a plane with normal (1, 1, 1) –Reduction of dimension –Similar to 3 D to 2 D in geometry – Each vector from (0, 0, 0) is an isochrominance line – Each vector maps to a point in the chromaticity chart 11

RGB-to-XYZ Space 12

CIE Chromaticity Chart • Shows all the visible colors • Achromatic Colors are at (0. 33, 0. 33) – Called white point • The saturated colors at the boundary – Spectral Colors 13

Chromaticity Chart: Hue • All colors on straight line from white point to a boundary has the same spectral hue – Dominant wavelength 14

Chromaticity Chart: Saturation • Purity (Saturation) – How far shifted towards the spectral color – Ratio of a/b – Purity =1 implies spectral color with maximum saturation 15

Color Reproducibility • Only a subset of the 3 D CIE XYZ space called 3 D color gamut • Projection of the 3 D color gamut –Triangle – 2 D color gamut Large if using more saturated primaries • Cannot describe brightness range reproducibility 16

Standard Color Gamut 17

Color spaces • RGB (CIE), Rn. Gn. Bn (TV – NTSC) • XYZ (CIE) • UVW (UCS de la CIE), U*V*W* (UCS modified by the CIE) • YUV, YIQ, YCb. Cr • HSV, HLS, IHS • Munsel colour space (cylindrical representation) • CIELuv • CIELab 18

RGB-to-YCb. Cr 19

Color Enhancement 20

Color Processing in the Compressed Domain Computation with reduced storage. Avoid overhead of inverse and forward transform. . Exploit spectral factorization for improving the quality of result and speed of computation. 21

Basic Approaches • Modify the DC coefficient for increasing brightness. Aghaglzadeh and Ersoy (1992), Opt. Engg. • Modify AC coefficients for increasing contrast. Tang, Peli and Acton (2003), IEEE SPL • A combination of both. S. Lee (2007), IEEE CSVT • Preserve also colors by processing DCT of chromatic components. 22

Different methods • Multi-Contrast Enhancement with Dynamic Range Compression (S. Lee (2007), IEEE CSVT) Modification of DC coefficients and AC coefficients (following similar strategy of multi-contrast enhancement). Normalized DC coefficients (x) are modified as follows: 23

Proposed Approach • Adjust background illumination. Use DC coefficients of the Y component. • Preserve Local Contrast. Scale AC coefficients of the Y component appropriately. • Preserve Colors. Preserve Color Vectors in the DCT domain. DCT coefficients of Cb and Cr components. 24

Contrast : Definition Let μ and σ denote the mean and standard deviation of an image. Contrast ζ of an image is defined here as: . Weber Law: where is the difference in luminance between a stimulus and its surround, and L is the luminance of the surround 25

Theorem on Contrast Preservation in the DCT Domain Let d be the scale factor for the DC coefficient and a a be the scale factor for the AC coefficients of a DCT block Y. The processed DCT block Ye is given by: The contrast of the processed image then becomes a / d times of the contrast of the original image. In this algorithm d = a = for preservation of the contrast. 26

Preservation of Colours in the DCT Domain Let U and V be the DCT coefficients of the Cb and Cr components, respectively. If the luminance component Y of an image is uniformly scaled by a factor , the colors of the processed image with Ye , Ue and Ve are preserved by the following operations: 27

Enhancement by Scaling Coefficients • Find the scale factor by mapping the DC coefficient with a monotonically increasing function. • Apply scaling to all other coefficients in all the components. • For blocks having greater details, apply block decomposition and re-composition strategy. 28

Mapping functions for adjusting the local background illumination (TW) (DRC) Mitra and Yu , CVGIP’ 87 Lee, CSVT’ 07 (SF) De, TENCON’ 89 29

Monotonic Mapping Functions 30

Scaling only DC coefficients 31

Scaling both DC and AC coefficients 32

Preservation of Contrast and Color original 33

Enhancement of Blocks with more details Block Decompos. 8 x 8 block Smaller DCT blocks Apply CES on smaller blocks Block Composition Enhanced Block 34

Removal of Blocking Artifacts original 35

Some Results original MCEDRC AR TW-CES-BLK MCE MSR 36

Enhancement near Edges AR TW-CES-BLK MCE DRC-CES-BLK MCEDRC SF-CES-BLK 37

Some Results original MCEDRC AR TW-CES-BLK MCE MSR 38

Enhancement near edges AR TW-CES-BLK MCE DRC-CES-BLK MCEDRC SF-CES-BLK 39

Some Results original MCEDRC AR TW-CES-BLK MCE MSR 40

Enhancement near edges AR TW-CES-BLK MCE DRC-CES-BLK MCEDRC SF-CES-BLK 41

Metrics for Comparison Wang and Bovic (SPL, 2002) JPEG Quality Metric (JPQM) Wang and Bovic (ICIP, 2002) Susstrunk and Winkler (SPIE, 2004) 42

Approaches under consideration • Alpha Rooting (AR) : Aghaglzadeh and Ersoy (1992), Opt. Engg. • Multi-Contrast Enhancement (MCE): Tang, Peli and Acton (2003), IEEE SPL • Multi-Contrast Enhancement with Dynamic Range Compression (MCEDRC): S. Lee (2007), IEEE CSVT • Contrast Enhancement by Scaling (CES): Proposed work • Multi-Scale Retinex (MSR) (a reference spatial domain technique): Jobson, Rahman and Woodell (1997), IEEE IP 43

Average Performance Measures Techniques AR MCEDRC TW-CESBLK DRC-CESBLK JPQM CEF Y- Cb- Cr. QM QM QM 8. 58 0. 97 0. 80 0. 67 7. 00 0. 94 0. 76 0. 67 7. 92 0. 97 0. 86 0. 67 7. 79 1. 50 0. 90 0. 82 0. 81 8. 16 1. 18 0. 86 0. 76 44

Computational Complexities Techniques Per Pixel Operations 1 E + 1 M AR 2. 19 M+1. 97 A MCE 0. 03 E+3. 97 M+2 A MCEDRC 0. 02 E+4. 02 M+1. 05 A TW-CES 0. 05 E+4 M+1. 08 A DRC-CES 0. 03 E+4. 02 M+1. 06 A SF-CES a. E+b. M+c. A implies a Exponentiation, b Multiplication and c Addition operations. 18 E+1866378 M+8156703 A MSR 45

Iterative Enhancement Iteration no. =1 Iteration no. =3 original Iteration no. =2 Iteration no. =4 46

Problem of Color Constancy • Three factors of image formation: Objects present in the scene. Spectral Energy of Light Sources. Spectral Sensitivity of sensors. Spectral Response of a Sensor Spectral Power Distribution Surface Reflectance Spectrum 47

Same Scene Captured under Different Illumination Can we transfer colors from one illumination to another one? 48

Computation of Color Constancy • Deriving an illumination independent representation. - Estimation of SPD of Light Source. E(λ) • Color Correction - Diagonal Correction. <R, G, B> 49 To perform this computation with DCT coefficients.

Different Spatial Domain Approaches • Gray World Assumption (Buchsbaum (1980), Gershon et al. (1988)) <R, G, B> ≡ <Ravg, Gavg, Bavg> • White World Assumption (Land (1977)) <R, G, B> ≡ <Rmax, Gmax, Bmax> 50

Select from a set of Canonical Illuminants Ø Observe distribution of points in 2 -D Chromatic Space. Ø Assign SPD of the nearest illuminant. • Gamut Mapping Approach (Forsyth (1990), Finlayson (1996)) - Existence of chromatic points. • Color by Correlation (Finlayson et. al. (2001)) - Relative strength over the distribution. • Nearest Neighbor Approach (Proposed) - Mean and Covariance Matrix. - Use of Mahalanobis Distance. 51

Processing in the Compressed Domain • Consists of non-overlapping DCT blocks (of 8 x 8). • Use DC coefficients of each block. • The color space used is Y-Cb-Cr instead of RGB. • Chromatic Space for Statistical Techniques is the Cb-Cr space. 52

Different Algorithms under consideration 53

List of Illuminants 54

Images Captured at Different Illumination Source: http: //www. cs. sfu. ca/ colour/data. 55

Performance Metrics • Estimated SPD: E=<RE, GE, BE> • True SPD: T= <RT, GT, BT> 56

Average Δθ 62

Average Δrg 63

Average ΔRGB 64

Average ΔL 65

Time and Storage Complexities • nl: number of illuminants. • nc: size of the 2 -D chromaticity space • n: number of image pixels • f: Fraction of chromaticity space covered. • a. M+b. A a number of Multiplications and b number of Additions. 66

Time and Storage Complexities 67

Equivalent No. of Additions per pixel (1 M= 3 A) n=512, nc=32, nl=12, f=1 68

Color Correction: An Example Image captured with (solux-4100) Target Ref. Image (syl-50 mr 16 q) MXW-DCT-Y COR-DCT 69

Color Restoration Original Enhanced w/o Color Correction Enhanced with Color Correction 70

Conclusion-I • Color-constancy computation in the compressed domain : - requires less time and storage. - comparable quality of results. • Both NN and NN-DCT perform well compared to other existing statistical approaches. • Color constancy computation is useful in restoration of colors. 71

Conclusion-II • Direct filtering in the 8 x 8 block DCT space using convolution multiplication properties. • Approximate and exact computations by block DCT composition and decomposition. • Demonstration of its applications in removing blocking artifacts and image enhancement. 72

Thanks 73