Image and Video Processing Dr V Kpuska Light

Image and Video Processing Dr. V. Këpuska

Light and Image Perception

Illumination u Quantification of human Psycho-visual sensation: n Peak spectral sensitivity of a human observer at 555 nm wavelength is normalized to unity. n Human spectral sensitivity drops to 4/10, 000 relative to peak at the two ends of the spectrum at 400 nm and 735 nm. 10/29/2020 Veton Këpuska 3

Visible Spectrum u We percieve electromagnetic energy having wavelengths in the range 400 -700 nm as visible light (slide taken from groups. csail. mit. edu/. . . /Lecture 02/spectrum. gif ). 10/29/2020 Veton Këpuska 4

Brightness u Brightness is proportional to logarithm of the luminous flux as defined by Fecher’s Law: u Consider a point source which emits luminous flux along radial lines: Ω – measured in steradians Ω 10/29/2020 r A Veton Këpuska 5

Luminous Intensity u Luminous Intensity I at a point source is defined as u Thus, luminous flux incident on area d. A from a source of intensity I is: 10/29/2020 Veton Këpuska 6

Illumination u The luminous flux per unit area falling on a surface is called illumination E and is given by: 10/29/2020 Veton Këpuska 7

Digital Image Representation

Digital Image Representation Image(y, x) ≡ Matrix(row, colum) 10/29/2020 Veton Këpuska 9

What is an Image u Collection of elements organized in a MATRIX form. u Each Matrix element represent smallest part of a picture element = pixel u Images are thus characterized by number of pixels n Larger the number of pixels the better the quality of image. n Larger the number of pixels the larger the storage space required to store the image. n More on this latter. 10/29/2020 Veton Këpuska 10

Representation of Pixels u What information is contained in pixels: n If the information is represented as a 4 -bit code than there could be only 24 = 16 different representations. n Typically 8 bits are used to represent each pixel. This defines a 28 = 256 levels of representation. u What does this level represent: n Gray Scale n Color – there are several ways to represent color that gives raise to various standards discusses next. 10/29/2020 Veton Këpuska 11

Pixel Information and its Representation u Gray Scale Example produced with MATLAB: u Two dimensional example from wikipedia with 212=4096 gray scale levels 10/29/2020 Veton Këpuska 12

Pixel Information and its Representation u Monochromatic Scale n The idea behind Gray scale image representation is to represent the image in a monochromatic scale. n Defined as “quality” of a single “color” u Color n From theory of light and theories how we perceive color information there a number of standards that have been developed to represent color. n They all address the Chromaticity – defined as the quality of a color as determined by its "purity" and dominant wavelength. 10/29/2020 Veton Këpuska 13

Color and How it is Perceived u In the study of the perception of color, one of the first mathematically defined color spaces was the CIE XYZ color space (also known as CIE 1931 color space), created by the International Commission on Illumination (CIE) in 1931. u The human eye has receptors for short (S), middle (M), and long (L) wavelengths, also known as blue, green, and receptors. That means that one, in principle, needs three parameters to describe a color sensation. A specific method for associating three numbers (or tristimulus values) with each color is called a color space: the CIE XYZ color space is one of many such spaces. However, the CIE XYZ color space is special, because it is based on direct measurements of the human eye, and serves as the basis from which many other color spaces are defined. The CIE XYZ color space was derived from a series of experiments done in the late 1920 s by W. David Wright (Wright 1928) and John Guild (Guild 1931). Their experimental results were combined into the specification of the CIE RGB color space, from which the CIE XYZ color space was derived. This article is concerned with both of these color spaces. u 10/29/2020 Veton Këpuska 14

Color and How it is Perceived u Since the human eye has three types of color sensor that respond to different ranges of wavelengths, a full plot of all visible colors is a three-dimensional figure. u However, the concept of color can be divided into two parts: brightness and chromaticity. For example, the color white is a bright color, while the color grey is considered to be a less bright version of that same white. In other words, the chromaticity of white and grey are the same while their brightness differs. u The CIE XYZ color space was deliberately designed so that the Y parameter was a measure of the brightness or luminance of a color. The chromaticity of a color was then specified by the two derived parameters x and y which are functions of all three tristimulus values X, Y, and Z: 10/29/2020 Veton Këpuska 15

Color and How it is Perceived 10/29/2020 Veton Këpuska 16

Color Model u A color model is an abstract mathematical model describing the way colors can be represented as tuples of numbers, typically as three or four values or color components (e. g. RGB and CMYK are color models). 10/29/2020 Veton Këpuska 17

Pixel Information and its Representation u The HSV (Hue, Saturation, Value) model, also known as HSB (Hue, Saturation, Brightness), defines a color space in terms of three constituent components: n Hue, the color type (such as red, blue, or yellow): u Ranges from 0 -360 (but normalized to 0 -100% in some applications) n Saturation, the "vibrancy" of the color: u Ranges from 0 -100% u Also sometimes called the "purity" by analogy to the colorimetric quantities excitation purity and colorimetric purity u The lower the saturation of a color, the more "grayness" is present and the more faded the color will appear, thus useful to define desaturation as the qualitative inverse of saturation n Value, the brightness of the color: u Ranges from 0 -100% 10/29/2020 Veton Këpuska 18

HSV-HSB Color Model u Example of HSV Color Model 10/29/2020 Veton Këpuska 19

RGB u u u RGB is shorthand for Red, Green, Blue. An RGB color space is any additive color space based on the RGB color model. RGB is a convenient color model for computer graphics because the human visual system works in a way that is similar—though not quite identical—to an RGB color space. The most commonly used RGB color spaces are s. RGB and Adobe RGB (which has a significantly larger gamut). Adobe has recently developed another color space called Adobe Wide Gamut RGB, which is even larger, in detriment to gamut density. As of 2007, s. RGB is by far the most commonly used RGB color space, particularly in consumer grade digital cameras, because it is considered adequate for most consumer applications, and its design simplifies previewing on the typical computer display. Adobe RGB is being built into more medium-grade digital cameras, and is favored by many professional graphic artists for its larger gamut. RGB spaces are generally specified by defining three primary colors and a white point. In the table below the three primary colors and white points for various RGB spaces are given. The primary colors are specified in terms of their CIE 1931 color space chromaticity coordinates (x, y). 10/29/2020 Veton Këpuska 20

RGB u RGB Color Space 10/29/2020 Veton Këpuska 21

CMYK u It is possible to achieve a large range of colors seen by humans by combining cyan, magenta, and yellow transparent dyes/inks on a white substrate. These are the subtractive primary colors. Often a fourth black is added to improve reproduction of some dark colors. This is called "CMY" or "CMYK" color space. u The cyan ink will reflect all but the red light, the yellow ink will reflect all but the blue light and the magenta ink will reflect all but the green light. This is because cyan light is an equal mixture of green and blue, yellow is an equal mixture of red and green, and magenta light is an equal mixture of red and blue. 10/29/2020 Veton Këpuska 22

CMYK u CMYK Color Space: 10/29/2020 Veton Këpuska 23

Color Image Representation u Wide Screen Computer Monitor: u 1920 x 1080 pixels n 1, 296, 000≈1, 3 M pixels u 32 -bit RGB Representation u Total: 1, 296, 000 x 32 x 3 = 124, 416, 000 ≈124 Mbits or u 24, 883, 200 ≈ 25 MBytes 10/29/2020 Veton Këpuska 24

Video Standards u NTSC - NTSC is the analog television system in use in Canada, Japan, South Korea, the Philippines, the United States, and some other countries, mostly in the Americas (see map). It is named for the National Television Standards Committee, the U. S. standardization body that adopted it. u PAL was developed by Walter Bruch at Telefunken in Germany. The format was first unveiled in 1963, with the first broadcasts beginning in the United Kingdom and Germany in 1967. [1] Telefunken was later bought by the French electronics manufacturer Thomson. u SECAM Compagnie Générale de Télévision where Henri de France developed SECAM, historically the first European colour television standard. 10/29/2020 Veton Këpuska 25

Video Standards u The term "PAL" is often used informally to refer to a 625 -line/50 Hz (576 i, principally European) television system, and to differentiate from a 525 -line/60 Hz (480 i, principally North American/Central American/Japanese) "NTSC" system. Accordingly, DVDs are labelled as either "PAL" or "NTSC" (referring informally to the line count and frame rate) even though technically neither of them have encoded PAL or NTSC composite colour. u The NTSC format is used with the M format (see broadcast television systems), which consists of 29. 97 interlaced frames of video per second. Each frame consists of 484 lines out of a total of 525 (the rest are used for sync, vertical retrace, and other data such as captioning). u PAL uses 625 lines, and so has a better picture quality. The NTSC system interlaces its scanlines, drawing odd-numbered scanlines in odd -numbered fields and even-numbered scanlines in even-numbered fields, yielding a nearly flicker-free image at its approximately 59. 94 hertz (nominally 60 Hz/100. 1%) refresh frequency. 10/29/2020 Veton Këpuska 26

Video Standards u NTSC’s refresh compares favorably to the 50 Hz refresh rate of the PAL and SECAM video formats used in Europe, where 50 Hz alternating current is the standard; flicker was more likely to be noticed when using these standards until modern PAL TV sets began using 100 Hz refresh rate to eliminate flicker. This produces a far more stable picture than native NTSC and PAL had, effectively displaying each frame twice. This did, at first, cause some motion problems, so it was not universally adopted until a few years ago. Interlacing the picture does complicate editing video, but this is true of all interlaced video formats, including PAL and SECAM. 10/29/2020 Veton Këpuska 27

Resolution u DVD images have 720 by 480 (NTSC) pixels or 720 by 576 (PAL) pixels 10/29/2020 Veton Këpuska 28

Image Processing Areas

Image Processing Areas u Acquisition, Storage and Retrieval n n n Digitization (sampling and quantization) Image compression Expansion of compressed images n n Image n n n Translation Scaling Rotation n n Comparison and Abstraction Classification u Enhancement equalization enhancement filtering De-noising u Transformation u Object Recognition 10/29/2020 Veton Këpuska 30

Geometric Transformations of an Image u Simple geometric transformations like translations, rotations and torsions are not trivial because of integer nature of the pixels 10/29/2020 Veton Këpuska 31

Image Transformations u Image Transformation can be done via Matrixes. u Matrixes transform vectors from one space to the other. n In geometry, an affine transformation or affine map (from the Latin, affinis, "connected with") between two vector spaces (strictly speaking, two affine spaces) consists of a linear transformation followed by a translation: 10/29/2020 Veton Këpuska 32

Image Transformations n This operation can be simplified by adopting the following notation for a generalized transformation matrix A and extension of input vector n This form is called homogenous coordinate system represention. 10/29/2020 Veton Këpuska 33

Examples of Transformation Matrixes u Translation: 10/29/2020 Veton Këpuska 34

Examples of Transformation Matrixes u Rotation: 10/29/2020 Veton Këpuska 35

Rotation Transformation 10/29/2020 Veton Këpuska 36

Scaling Transformation u Scaling 10/29/2020 Veton Këpuska 37

Shearing Transformation u Shear parallel to x axis 10/29/2020 Veton Këpuska 38

Shearing Transformation u Shear parallel to y axis 10/29/2020 Veton Këpuska 39

Reflection Transformation u Reflection: To reflect a vector about a line that goes through the origin, let (ux, uy) be a unit vector in the direction of the line; then: 10/29/2020 Veton Këpuska 40

Example Using “Lena” 10/29/2020 Veton Këpuska 41

Image Enhancement

Image Equalization u Histogram Based Equalization n Any gray-level modification technique is based on creating a mapping of the gray levels in the original image to the gray levels in the modified image. n Let x represent the original image as a gray-level. n Let y represent the modified image. y=T(x) 10/29/2020 Veton Këpuska 43

Histogram Flattening u Histogram Computation u PDF of the histogram for the image of the size of Mx. N u CDF from the PDF 10/29/2020 Veton Këpuska 44

Histogram Flattening u Transformation Mapping 10/29/2020 Veton Këpuska 45

Example: Lena Original Image cd f pdf Trans 10/29/2020 Veton Këpuska 46

Linear Filtering of Images u Definition of 2 D-Linear Filtering: n It is the operation that it is applied to the image x(k, l) to obtain the image y(k, l) defined by: n The sequence h(k, l) dependent on to indexes that defines a filter, is called the Point Spread Function (PSF). 10/29/2020 Veton Këpuska 47

Linear Filtering of Images u As was the case with 1 D (e. g. , sequences of time) filtering n the convolution operation is linear and space-invariant. n h(k, l) is the equivalent of the impulse response of the one dimensional case. 10/29/2020 Veton Këpuska 48

Convolution – Multiplication Equivalence u If x, h and y are related by u The X, H and Y the Discrete Frequency Representations are related by u X, H and Y are 2 D DFTs of x, h, and y respectively. 10/29/2020 Veton Këpuska 49

Separable Filters u A filter is said to be separable if its PSF has the following property: u In case of finite PSF represented by matrix h of elements h(k, l) and if hx and hy are the vectors with respective components hx(k) and hy(l) then the relation above is equivalent to: 10/29/2020 Veton Këpuska 50

Separability and 1 D Filtering 10/29/2020 Veton Këpuska 51

The Guassian Filter is Separable 10/29/2020 Veton Këpuska 52

Gaussian Smoothing Filter u Gaussian Kernel 10/29/2020 Veton Këpuska 53

Gaussian PSF Kernel 10/29/2020 Veton Këpuska 54

Low Pass Filtering Example u Filter Mask g 2 f = 0. 0000 0. 0001 0. 0002 0. 0001 0. 0000 0. 0002 0. 0005 0. 0002 0. 0000 0. 0001 0. 0004 0. 0011 0. 0004 0. 0001 0. 0002 0. 0006 0. 0018 0. 0006 0. 0002 0. 0008 0. 0024 0. 0008 0. 0002 0. 0009 0. 0026 0. 0009 0. 0002 0. 0008 0. 0024 0. 0008 0. 0002 0. 0006 0. 0018 0. 0006 0. 0002 0. 0001 0. 0004 0. 0011 0. 0004 0. 0001 0. 0000 0. 0002 0. 0005 0. 0002 0. 0000 0. 0001 0. 0002 0. 0001 0. 0000 0. 0000 10/29/2020 0. 0000 0. 0001 0. 0002 0. 0001 0. 0000 0. 0002 0. 0004 0. 0006 0. 0008 0. 0009 0. 0008 0. 0006 0. 0004 0. 0002 0. 0001 0. 0005 0. 0011 0. 0018 0. 0024 0. 0026 0. 0024 0. 0018 0. 0011 0. 0005 0. 0002 0. 0013 0. 0026 0. 0043 0. 0058 0. 0064 0. 0058 0. 0043 0. 0026 0. 0013 0. 0005 0. 0026 0. 0053 0. 0087 0. 0117 0. 0129 0. 0117 0. 0087 0. 0053 0. 0026 0. 0011 0. 0043 0. 0087 0. 0143 0. 0193 0. 0213 0. 0193 0. 0143 0. 0087 0. 0043 0. 0018 0. 0058 0. 0117 0. 0193 0. 0261 0. 0288 0. 0261 0. 0193 0. 0117 0. 0058 0. 0024 0. 0064 0. 0129 0. 0213 0. 0288 0. 0318 0. 0288 0. 0213 0. 0129 0. 0064 0. 0026 0. 0058 0. 0117 0. 0193 0. 0261 0. 0288 0. 0261 0. 0193 0. 0117 0. 0058 0. 0024 0. 0043 0. 0087 0. 0143 0. 0193 0. 0213 0. 0193 0. 0143 0. 0087 0. 0043 0. 0018 0. 0026 0. 0053 0. 0087 0. 0117 0. 0129 0. 0117 0. 0087 0. 0053 0. 0026 0. 0011 0. 0013 0. 0026 0. 0043 0. 0058 0. 0064 0. 0058 0. 0043 0. 0026 0. 0013 0. 0005 0. 0011 0. 0018 0. 0024 0. 0026 0. 0024 0. 0018 0. 0011 0. 0005 0. 0002 0. 0004 0. 0006 0. 0008 0. 0009 0. 0008 0. 0006 0. 0004 0. 0002 0. 0001 0. 0000 0. 0001 0. 0002 Veton Këpuska 0. 0002 0. 0001 0. 0000 55

Filter Mask 10/29/2020 Veton Këpuska 56

1. Moving Average (Low Pass) 10/29/2020 Veton Këpuska 57

Moving Average 1. Sobel Filter (Edge Detection – High Pass) 10/29/2020 Veton Këpuska 58

1. Laplacian Filter (High Pass) 10/29/2020 Veton Këpuska 59

Averaging Filter 10/29/2020 Veton Këpuska 60

Sobel Filter 10/29/2020 Veton Këpuska 61

Laplacean Filter 10/29/2020 Veton Këpuska 62

Median Filtering u The original pixel is replaced by the median of the pixels contained in the neighborhood: 10/29/2020 Veton Këpuska 63

Visual. DSP++ Examples u Color Conversion Schemes n RGB to/from NTSC format u MATLAB: rgb 2 ntsc & ntsc 2 rgb 10/29/2020 Veton Këpuska 64

Visual. DSP++ Examples u Loading Images: n View ➙ Debug Windows ➙ Image Viewer n Image Configuration Window: u Source Location of the image file: n http: //my. fit. edu/~vkepuska/ece 3552/esp_boo k/adsp/chap 10/image_files/Disney. jpg n http: //my. fit. edu/~vkepuska/ece 3552 u Start Address: n 0. 0 (Address of SDRAM of EZ-Kit Lite) u Memory Stride n 1 10/29/2020 Veton Këpuska 65

Visual. DSP++ Examples u Color Conversion: n Copy Color Conversion Projects from http: //my. fit. edu/~vkepuska/ece 3552/esp_book/ads p/chap 10/exp 10_2/ u rgb 2 gray u rgb 2 ntsc n Build and run the project and compare the result to MATLAB simulation. 10/29/2020 Veton Këpuska 66

Visual. DSP++ Examples u 2 D Filtering: n Copy Filtering Projects from http: //my. fit. edu/~vkepuska/ece 3552/esp_book/adsp/chap 10/ exp 10_4/ u exp 10_4. dpj n n n Build and run the project. Observe the image from the starting address 0 x 130000. Implementation Details: u Image is stored as unsigned char (8 -bit data) type. u 2 D convolution routine is implemented with Q 1. 15 16 -bit fixed point format. u Input 8 -bit data scaled to 16 -bit by multiplication with a constant of 0 x 80. u Processed image must be scaled back to 8 -bit by dividing with 0 x 80. 10/29/2020 Veton Këpuska 67

Enhancement of the Previous Lab Exercise u Follow the provided link to obtain Video Library n http: //my. fit. edu/~vkepuska/ece 3552/Video. Library/ u Instructions on how to use the library are given at: n http: //www 2. enel. ucalgary. ca/People/Smith/ECE-ADI -Project/Blackfin. Video. Libraries/index. html u Extend Previous Project by linking the Video Libraries and displaying the original and processed image on the TV (after being converted to NTSC format). u For further information on video processing see this link n http: //www 2. enel. ucalgary. ca/People/Smith/ECE-ADI -Project/Blackfin. Video. Libraries/index. html 10/29/2020 Veton Këpuska 68