Color Computer Vision Martin Jagersand What is the

Color Computer Vision Martin Jagersand What is the physics of Readings: color? How is color sensed? –Szeliski, 2. 3. 2 How is it represented? –Forsyth and Ponce, Chapter 6 Applications in vision?

Physics: Light = Electro Magnetic Radiation Color = Wavelength

Sunlight White sunlight: composed of about equal energy in all wavelengths of the visible spectrum Newton 1665 Image from http: //micro. magnet. fsu. edu/

Sunlight

The Physics of Light Rel. power Some examples of the spectra of light sources © Stephen E. Palmer, 2002

For a surface Color = response curve What is the inherent dimensionality of color space ?

For a Human • Color is a psychological property of our visual experiences when we look at objects and lights, not a physical property of those objects or lights (S. Palmer, Vision Science: Photons to Phenomenology) • Color is the result of interaction between physical light in the environment and our visual system Wassily Kandinsky (1866 -1944), Murnau Street with Women, 1908

Biological vision and color constancy • One of the most shared images on facebook &Twitter

Photo receptor distribution in the human eye

Color perception 100 212 50 42 175 66 244 255 31 0 196 Human RGB Reponses R 115 G 47 B 28 What happens as the intensity of the light changes?

Color receptors “Red” cone Response of k’th cone = “Green” cone “Blue” cone Principle of univariance: cones give the same kind of response, in different amounts, to different wavelengths. Output of cone is obtained by summing over wavelengths. Responses measured in a variety of ways

RGB response Response curves are overlapping ! b(l) g(l) Response of each R, G or B is integral of incoming light with sensitivity function r(l)

Metamers • Two different Spectral Energy Distributions with the same RED, GREEN, BLUE response are termed metamers. b(l) g(l) r(l) Radiance (Energy) Wavelength l

Color mixing Cartoon spectra for color names: Source: W. Freeman

Additive color mixing Colors combine by adding color spectra + = Light is added to black. Source: W. Freeman

Examples of additive color systems CRT phosphors multiple projectors http: //www. jegsworks. com http: //www. crtprojectors. co. uk/

Subtractive color mixing - Colors combine by multiplying color spectra. = Pigments remove color from incident light (white). Source: W. Freeman

Why specify color numerically? • Accurate color reproduction is commercially valuable – Many products are identified by color (“golden” arches) • Few color names are widely recognized by English speakers 11: black, blue, brown, grey, green, orange, pink, purple, red, white, and yellow. – Other languages have fewer/more. – Common to disagree on appropriate color names. – • Color reproduction problems increased by prevalence of digital imaging – e. g. digital libraries of art. How to ensure that everyone perceives the same color? – What spectral radiances produce the same response from people under simple viewing conditions? – Forsyth & Ponce

Standard Color System RED RGB is a standard system, but is formally defined in terms of the CIE system. magenta yellow BLUE GREEN RGB cyan

Filtering Colors Short wavelength Long wavelength

Color Constancy How do humans adapt to the variability of color?

Color Constancy Recognizing color differences

Color Constancy: Land’s demonstration

Land’s Demonstration

Other color spaces: YUV or YIQ • Invented for color television • Backward compatible with B/W TV • Y given higher bandwidth than I/Q

Other color spaces: HSI INTENSITY red magenta saturation yellow hue blue green cyan

HSI: Factoring out intensity I = ( R + G + B )/3 S = ( 1 - min (R, G, B)/ I ) H = 0 + (G-B)/ if max is R = 1/3 + (B-R)/ if max is G = 2/3 + (R-G)/ if max is B H S ( is (max-min) of RGB) HSI I

Summary: Other Color Spaces • RGB: sensor-based description • HSI: axes better aligned with intrinsic surface reflectance • CMY, CMYK: subtractive color, used in printers C = 1 -R M = 1 -G Y = 1 -B • YIQ : television, YUV: used in JPEG

Homogeneous Region Sample PCA-fitted ellipsoid

Homogeneous Color Region: Photometry

Applications Chromakeying • Margaret Fleck’s work on image filtering • David Forsyth’s work on color constancy Artists’ palette

Color histograms Swain and Ballard, Color Indexing, IJCV 1991. • Color histograms for indexing and retrieval uses opponent-axes Tested on product packaging: “The only target for which the effectiveness of Histogram Backprojection suffers badly is that of Charmin paper towels”

Color Histograms model feature vectors Test 1 RGB A bit of binning. . . Test 2

Histogram Metrics • Intersection min(Mi , Ti) i i will be greater than Ti • Distance (can be weighted) (Mi - Ti)’ W (Mi - Ti) will be smaller than 1 0 0 0 0 00 1 0. 5 0 1 0 00 0 0 0 0 1 0 0 0 0 1

Color-based image retrieval • Given collection (database) of images: – Extract and store one color histogram per image • Given new query image: Extract its color histogram – For each database image: – – Compute intersection between query histogram and database histogram Sort intersection values (highest score = most similar) – Rank database items relative to query based on this sorted order –

Color-based image retrieval Example database

Color-based image retrieval Example retrievals

Color-based image retrieval Example retrievals

Histogram Backprojection For each local histogram of the appropriate size, give its location a weighted histogram intersection score. i wi min(Mi , Li) model Image full image's histogram Problems? Image local histogram from I wi = 1/Ii Efficiency? model

Radiometry for color • All definitions are now “per unit wavelength” • All units are now “per unit wavelength” • All terms are now “spectral” • Radiance becomes spectral radiance – watts per square meter per steradian per unit wavelength • Radiosity --- spectral radiosity

RADIANCE VS. LUMINANCE RADIANCE 1 Watt/(meter^2 x steradian x cosq) LUMINANCE = 683 Lumens at 555 nm Luminous Efficacy

SPECTRAL ENERGY DISTRIBUTION AND PHOTOPIC LUMINANCE I(l) Radiance (Energy) Human sensitivity curve P(l) Wavelength l