CS 414 Multimedia Systems Design Lecture 4 Visual

CS 414 – Multimedia Systems Design Lecture 4 – Visual Perception and Digital Image Representation Klara Nahrstedt Spring 2014 CS 414 - Spring 2014

Administrative Groups are formed and names have been sent to Engineering IT and Barb Leisner n We will inform you about group directories as soon as we have information from Engineering IT n CS 414 - Spring 2014

Administrative n Leasing Process from Barb Leisner ¨ ¨ ¨ Lease one Logitech camera - two cameras within one group to start MP 1, and then for MP 2/MP 3. Leasing process starts on January 31 Pick up the camera from Barb Leisner office, 2312 SC Bring your student ID to sign for the camera Each cs 414 group is responsible for his/her own camera n if you loose it (or badly damage) and you don’t have police report, you pay for it (charged to your student account at the end of the semester) Hours to pick up camera: Monday –Friday 9 am-5 pm ¨ No camera pickup on Saturday and Sunday ¨ CS 414 - Spring 2014

Today Introduced Concepts n Important Metric for Digital Audio ¨ Signal-to-Noise Ratio (d. B) Human Visual System n Digital Images n ¨ Sampling ¨ Quantization ¨ Spatial Resolution CS 414 - Spring 2014

Signal-to-Noise Ratio (metric to quantify quality of digital audio) CS 414 - Spring 2014

Signal To Noise (SNR) Ratio n Measures strength of signal to noise SNR (in DB)= n Given sound form with amplitude in [-A, A] A n Signal energy = 0 -A CS 414 - Spring 2014

Modeling of Noise Quantization Error n Difference between actual and sampled value ¨ amplitude between [-A, A] ¨ quantization levels = N n e. g. , if A = 1, N = 8, = 1/4 CS 414 - Spring 2014

Compute Signal to Noise Ratio n Signal energy = n Noise energy = n Signal-to-Noise = n SNR depends on number of bits (number of quantization levels) assigned to signal Every bit increases SNR by ~ 6 decibels n ; Noise energy = ;

Integrating Aspects of Multimedia Image/Video Capture Audio/Video Perception/ Playback Audio/Video Presentation Playback Image/Video Information Representation Transmission Audio Capture Transmission Compression Processing Audio Information Representation Media Server Storage CS 414 - Spring 2014 A/V Playback

Human Visual System n Eyes, optic nerve, parts of the brain n Transforms electromagnetic energy

Human Visual System n Image Formation ¨ cornea, sclera, pupil, iris, lens, retina, fovea n Transduction ¨ retina, rods, and cones ¨ Retina has photosensitive receptors at back of eye n Processing ¨ optic nerve, brain

Rods vs Cones (Responsible for us seeing brightness and color) Cones n n n Rods Contain photo-pigment Respond to high energy Enhance perception Concentrated in fovea, exist sparsely in retina Three types, sensitive to different wavelengths n n n Contain photo-pigment Respond to low energy Enhance sensitivity Concentrated in retina, but outside of fovea One type, sensitive to grayscale changes CS 414 - Spring 2014

Tri-stimulus Theory n 3 types of cones (6/7 Mil. of them) Red = L cones, Green = M cones, Blue = S cones ¨ Ratio differentiates for each person ¨ E. g. , Red (64%), Green (32%), rest S cones ¨ E. g. , L(50. 6%), M(44. 2%), rest S cones ¨ n Each type most responsive to a narrow band electro-magnetic waves ¨ n red and green absorb most energy, blue the least Light stimulates each set of cones differently, and the ratios produce CS 414 - Spring 2014 sensation of color

Color and Visual System n Color refers to how we perceive a narrow band of electromagnetic energy ¨ source, object, observer n Visual system transforms light energy into sensory experience of sight

Color Perception (Color Theory) n Hue Scale ¨ Refers to pure colors ¨ dominant wavelength of the light Saturation ¨ Perceived n Brightness (lightness) Source: Wikipedia lightness ¨ perceived Saturation intensity of a specific color ¨ how far color is from a gray of equal intensity Original n intensity CS 414 - Spring 2014

Digitalization of Images – Capturing and Processing CS 414 - Spring 2014

Capturing Real-World Images n Picture – two dimensional image captured from a real-world scene that represents a momentary event from the 3 D spatial W 2 world W 1 r W 3 F r= function of (W 1/W 3); s=function of (W 2/W 3) s CS 414 - Spring 2014

Image Concepts - Sampling An image is a function of intensity values over a 2 D plane I(r, s) n Sample function at discrete intervals to represent an image in digital form n ¨ matrix of intensity values for each color plane ¨ intensity typically represented with 8 bits n Sample points are called pixels CS 414 - Spring 2014

Digital Image Sampling n Sample = pixel n Image Size (in pixels) n Image Size = Height x Width (in pixels) 320 x 240 pixels n 640 x 480 pixels n 1920 x 1080 pixels n CS 414 - Spring 2014

Digital Images - Quantization = number of bits per pixel n Example: if we would sample and quantize standard TV picture (525 lines) by using VGA (Video Graphics Array), n ¨ video controller creates matrix 640 x 480 pixels, and ¨ each pixel is represented by 8 bit integer (256 discrete gray levels) CS 414 - Spring 2014

Image Representations n Black and white image ¨ single bits n Grey scale image ¨ single bits n color plane with 2 color plane with 8 Color image ¨ three color planes each with 8 bits ¨ RGB, CMY, YIQ, etc. n Indexed color image ¨ single plane that indexes a color table n Compressed images ¨ TIFF, JPEG, BMP, etc. 4 gray levels 2 gray levels

Digital Image Representation (3 Bit Quantization) CS 414 - Spring 2014

Color Quantization Example of 24 bit RGB Image 24 -bit Color Monitor CS 414 - Spring 2014

Image Representation Example 24 bit RGB Representation (uncompressed) 128 135 166 138 190 132 129 255 105 189 167 190 229 213 134 111 138 187 135 255 213 190 167 138 129 229 138 189 111 Color Planes 166 105 134 132 190 187

Graphical Representation CS 414 - Spring 2014

Image Properties (Color) CS 414 - Spring 2014

Color Histogram CS 414 - Spring 2014

Spatial and Frequency Domains n Spatial domain ¨ refers to planar region of intensity values at time t n Frequency domain ¨ think of each color plane as a sinusoidal function of changing intensity values ¨ refers to organizing pixels according to their changing intensity (frequency) CS 414 - Spring 2014

Spatial Resolution and Brightness n Spatial Resolution (depends on: ) Image size ¨ Viewing distance ¨ n Brightness Perception of brightness is higher than perception of color ¨ Different perception of primary colors n Relative brightness: green: red: blue= ¨ 59%: 30%: 11% CS 414 - Spring 2014 Source: wikipedia

Image Size (in Bits) n Image Size = Height x Width X Bits/pixel n Example: ¨ Consider image 320 x 240 pixels with 8 bits per pixel ¨ Image takes storage 7680 x 8 bits = 61440 bits or 7680 bytes CS 414 - Spring 2014

Summary n Important Image Processing Functions (see Computer Vision/Image Processing classes) ¨ Filtering ¨ Edge detection ¨ Image segmentation ¨ Image recognition n n n ¨ Formatting Conditioning Marking Grouping Extraction Matching Image synthesis CS 414 - Spring 2014