082715 Pixels and Image Filtering Computational Photography Derek

08/27/15 Pixels and Image Filtering Computational Photography Derek Hoiem Graphic: http: //www. notcot. org/post/4068/

Administrative stuff • Any questions? • Office hours: times will be set Friday, so be sure to fill out poll http: //doodle. com/8 cqbis 4 qx 52 wm 5 fs • Tutorial: • Looks like Sept 2 at 5 pm, but will finalize later today --- vote soon http: //doodle. com/6 drhrg 3 kdedu 892 x

Today’s Class: Pixels and Linear Filters • What is a pixel? How is an image represented? • What is image filtering and how do we do it? • Introduce Project 1: Hybrid Images

Next three classes • Image filters in spatial domain – Smoothing, sharpening, measuring texture • Image filters in the frequency domain – Denoising, sampling, image compression • Templates and Image Pyramids – Detection, coarse-to-fine registration

Image Formation

Digital camera A digital camera replaces film with a sensor array • • Each cell in the array is light-sensitive diode that converts photons to electrons Two common types: – Charge Coupled Device (CCD): larger yet slower, better quality – Complementary Metal Oxide Semiconductor (CMOS): high bandwidth, lower quality • http: //electronics. howstuffworks. com/digital-camera. htm Slide by Steve Seitz

Sensor Array CMOS sensor CCD sensor

The raster image (pixel matrix)

The raster image (pixel matrix) 0. 92 0. 95 0. 89 0. 96 0. 71 0. 49 0. 86 0. 96 0. 69 0. 79 0. 91 0. 93 0. 89 0. 72 0. 95 0. 81 0. 62 0. 84 0. 67 0. 49 0. 73 0. 94 0. 82 0. 51 0. 88 0. 81 0. 60 0. 74 0. 56 0. 90 0. 89 0. 97 0. 89 0. 55 0. 94 0. 87 0. 58 0. 85 0. 66 0. 67 0. 49 0. 62 0. 56 0. 51 0. 56 0. 57 0. 50 0. 51 0. 48 0. 43 0. 33 0. 41 0. 37 0. 31 0. 42 0. 46 0. 37 0. 60 0. 39 0. 37 0. 42 0. 61 0. 78 0. 85 0. 75 0. 57 0. 91 0. 80 0. 58 0. 73 0. 88 0. 77 0. 69 0. 78 0. 97 0. 92 0. 41 0. 87 0. 88 0. 50 0. 92 0. 90 0. 73 0. 79 0. 77 0. 93 0. 81 0. 49 0. 90 0. 89 0. 61 0. 94 0. 71 0. 73 0. 89 0. 92 0. 95 0. 91 0. 97 0. 79 0. 45 0. 49 0. 82 0. 90 0. 93 0. 99 0. 91 0. 92 0. 95 0. 85 0. 33 0. 74 0. 93 0. 99 0. 97 0. 93

Perception of Intensity from Ted Adelson

Perception of Intensity from Ted Adelson

Digital Color Images

Color Image R G B

Images in Matlab • Images represented as a matrix • Suppose we have a Nx. M RGB image called “im” – im(1, 1, 1) = top-left pixel value in R-channel – im(y, x, b) = y pixels down, x pixels to right in the bth channel – im(N, M, 3) = bottom-right pixel in B-channel • imread(filename) returns a uint 8 image (values 0 to 255) – Convert to double format (values 0 to 1) with im 2 double row column 0. 92 0. 95 0. 89 0. 96 0. 71 0. 49 0. 86 0. 96 0. 69 0. 79 0. 91 0. 93 0. 89 0. 72 0. 95 0. 81 0. 62 0. 84 0. 67 0. 49 0. 73 0. 94 0. 82 0. 51 0. 92 0. 88 0. 95 0. 81 0. 89 0. 60 0. 96 0. 74 0. 71 0. 54 0. 49 0. 56 0. 86 0. 90 0. 96 0. 89 0. 69 0. 79 0. 91 0. 97 0. 89 0. 55 0. 93 0. 94 0. 89 0. 87 0. 72 0. 58 0. 95 0. 58 0. 81 0. 85 0. 62 0. 66 0. 84 0. 67 0. 49 0. 73 0. 94 0. 62 0. 56 0. 51 0. 94 0. 56 0. 82 0. 57 0. 51 0. 92 0. 50 0. 88 0. 95 0. 51 0. 89 0. 48 0. 60 0. 96 0. 43 0. 74 0. 71 0. 33 0. 54 0. 49 0. 41 0. 56 0. 86 0. 90 0. 96 0. 89 0. 69 0. 79 0. 91 0. 37 0. 31 0. 42 0. 97 0. 46 0. 89 0. 37 0. 55 0. 93 0. 60 0. 94 0. 89 0. 39 0. 87 0. 72 0. 37 0. 58 0. 95 0. 42 0. 58 0. 81 0. 61 0. 85 0. 62 0. 78 0. 66 0. 84 0. 67 0. 49 0. 73 0. 94 0. 85 0. 75 0. 57 0. 62 0. 91 0. 56 0. 80 0. 51 0. 94 0. 58 0. 56 0. 82 0. 73 0. 57 0. 51 0. 88 0. 50 0. 88 0. 77 0. 51 0. 81 0. 69 0. 48 0. 60 0. 78 0. 43 0. 74 0. 33 0. 54 0. 41 0. 56 0. 90 0. 89 0. 97 0. 92 0. 41 0. 37 0. 87 0. 31 0. 88 0. 42 0. 97 0. 50 0. 46 0. 89 0. 92 0. 37 0. 55 0. 90 0. 60 0. 94 0. 73 0. 39 0. 87 0. 79 0. 37 0. 58 0. 77 0. 42 0. 58 0. 61 0. 85 0. 78 0. 66 0. 67 0. 49 0. 93 0. 81 0. 49 0. 85 0. 90 0. 75 0. 89 0. 57 0. 62 0. 61 0. 91 0. 56 0. 91 0. 80 0. 51 0. 94 0. 58 0. 56 0. 71 0. 73 0. 57 0. 73 0. 88 0. 50 0. 89 0. 77 0. 51 0. 69 0. 48 0. 78 0. 43 0. 33 0. 41 0. 92 0. 95 0. 91 0. 97 0. 92 0. 79 0. 41 0. 37 0. 45 0. 87 0. 31 0. 49 0. 88 0. 42 0. 82 0. 50 0. 46 0. 90 0. 92 0. 37 0. 93 0. 90 0. 60 0. 99 0. 73 0. 39 0. 79 0. 37 0. 77 0. 42 0. 61 0. 78 0. 99 0. 91 0. 92 0. 93 0. 95 0. 81 0. 85 0. 49 0. 85 0. 33 0. 90 0. 75 0. 74 0. 89 0. 57 0. 93 0. 61 0. 99 0. 91 0. 80 0. 97 0. 94 0. 58 0. 93 0. 71 0. 73 0. 88 0. 89 0. 77 0. 69 0. 78 R 0. 92 0. 95 0. 91 0. 97 0. 92 0. 79 0. 41 0. 45 0. 87 0. 49 0. 88 0. 82 0. 50 0. 92 0. 93 0. 90 0. 99 0. 73 0. 79 0. 77 0. 99 0. 91 0. 92 0. 93 0. 95 0. 81 0. 85 0. 49 0. 33 0. 90 0. 74 0. 89 0. 93 0. 61 0. 99 0. 91 0. 97 0. 94 0. 93 0. 71 0. 73 0. 89 G B 0. 92 0. 95 0. 91 0. 97 0. 79 0. 45 0. 49 0. 82 0. 90 0. 93 0. 99 0. 91 0. 92 0. 95 0. 85 0. 33 0. 74 0. 93 0. 99 0. 97 0. 93

Image filtering • Image filtering: compute function of local neighborhood at each position • Really important! – Enhance images • Denoise, resize, increase contrast, etc. – Extract information from images • Texture, edges, distinctive points, etc. – Detect patterns • Template matching

Example: box filter 1 1 1 1 1 Slide credit: David Lowe (UBC)

Image filtering 0 0 0 0 0 0 90 90 90 0 0 90 90 90 0 0 0 0 90 90 90 0 0 90 90 90 0 0 0 0 0 0 0 90 90 0 0 0 0 0 0 0 0 1 1 1 1 1 0 Credit: S. Seitz

Image filtering 0 0 0 0 0 0 90 90 90 0 0 90 90 90 0 90 90 90 0 0 0 0 0 0 0 0 0 1 1 10 Credit: S. Seitz

Image filtering 0 0 0 0 0 0 90 90 90 0 0 90 90 90 0 90 90 90 0 0 0 0 0 0 0 0 0 10 1 1 1 1 1 20 Credit: S. Seitz

Image filtering 0 0 0 0 0 0 90 90 90 0 0 90 90 90 0 90 90 90 0 0 0 0 0 0 0 0 0 10 20 1 1 1 1 1 30 Credit: S. Seitz

Image filtering 0 0 0 0 0 0 90 90 90 0 0 90 90 90 0 90 90 90 0 0 0 0 0 0 0 0 0 10 20 30 1 1 1 1 1 30 Credit: S. Seitz

Image filtering 0 0 0 0 0 0 90 90 90 0 0 90 90 90 0 90 90 90 0 0 0 0 0 0 0 0 0 10 20 30 1 1 1 1 1 30 ? Credit: S. Seitz

Image filtering 0 0 0 0 0 0 90 90 90 0 0 90 90 90 0 90 90 90 0 0 0 0 0 0 0 0 0 10 20 30 1 1 1 1 1 30 ? 50 Credit: S. Seitz

Image filtering 1 1 1 1 1 0 0 0 0 0 0 10 20 30 30 30 20 10 0 90 90 90 0 20 40 60 60 60 40 20 0 90 90 90 0 30 60 90 90 90 60 30 0 90 90 90 0 30 50 80 80 90 60 30 0 90 90 90 0 20 30 50 50 60 40 20 0 0 10 20 30 30 20 10 0 0 90 0 0 0 10 10 10 0 0 0 Credit: S. Seitz

Image filtering 1 1 1 1 1 0 0 0 0 0 0 10 20 30 30 30 20 10 0 90 90 90 0 20 40 60 60 60 40 20 0 Informally, what does the filter do? 0 0 90 90 90 0 30 60 90 90 90 60 30 0 0 90 90 90 0 30 50 80 80 90 60 30 0 90 90 90 0 20 30 50 50 60 40 20 0 0 10 20 30 30 20 10 0 0 90 0 0 0 10 10 10 0 0 0 0 Credit: S. Seitz

Box Filter What does it do? • Replaces each pixel with an average of its neighborhood • Achieve smoothing effect (remove sharp features) 1 1 1 1 1 Slide credit: David Lowe (UBC)

Smoothing with box filter

One more on board…

Practice with linear filters 0 0 1 0 0 ? Original Source: D. Lowe

Practice with linear filters Original 0 0 1 0 0 Filtered (no change) Source: D. Lowe

Practice with linear filters 0 0 0 1 0 0 0 ? Original Source: D. Lowe

Practice with linear filters Original 0 0 0 1 0 0 0 Shifted left By 1 pixel Source: D. Lowe

Practice with linear filters 0 0 2 0 0 - 1 1 1 1 1 ? (Note that filter sums to 1) Original Source: D. Lowe

Practice with linear filters Original 0 0 2 0 0 - 1 1 1 1 1 Sharpening filter - Accentuates differences with local average Source: D. Lowe

Sharpening Source: D. Lowe

Other filters 1 0 -1 2 0 -2 1 0 -1 Sobel Vertical Edge (absolute value)

Q? Other filters 1 2 1 0 0 0 -1 -2 -1 Sobel Horizontal Edge (absolute value)

How could we synthesize motion blur? theta = 30; len = 20; fil = imrotate(ones(1, len), theta, 'bilinear'); fil = fil / sum(fil(: )); figure(2), imshow(imfilter(im, fil));

Filtering vs. Convolution • 2 d filtering g=filter f=image – h=filter 2(g, f); or h=imfilter(f, g); • 2 d convolution – h=conv 2(g, f);

Key properties of linear filters Linearity: filter(f 1 + f 2) = filter(f 1) + filter(f 2) Shift invariance: same behavior regardless of pixel location filter(shift(f)) = shift(filter(f)) Any linear, shift-invariant operator can be represented as a convolution Source: S. Lazebnik

More properties • Commutative: a * b = b * a – Conceptually no difference between filter and signal (image) • Associative: a * (b * c) = (a * b) * c – Often apply several filters one after another: (((a * b 1) * b 2) * b 3) – This is equivalent to applying one filter: a * (b 1 * b 2 * b 3) • Distributes over addition: a * (b + c) = (a * b) + (a * c) • Scalars factor out: ka * b = a * kb = k (a * b) • Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a Source: S. Lazebnik

Important filter: Gaussian • Weight contributions of neighboring pixels by nearness 0. 003 0. 013 0. 022 0. 013 0. 003 0. 013 0. 059 0. 097 0. 059 0. 013 0. 022 0. 097 0. 159 0. 097 0. 022 0. 013 0. 059 0. 097 0. 059 0. 013 0. 003 0. 013 0. 022 0. 013 0. 003 5 x 5, = 1 Slide credit: Christopher Rasmussen

Smoothing with Gaussian filter

Smoothing with box filter

Gaussian filters • Source: K. Grauman

Separability of the Gaussian filter Source: D. Lowe

Separability example 2 D filtering (center location only) The filter factors into a product of 1 D filters: Perform filtering along rows: Followed by filtering along the remaining column: * = Source: K. Grauman

Separability • Why is separability useful in practice?

Some practical matters

Practical matters How big should the filter be? • Values at edges should be near zero • Rule of thumb for Gaussian: set filter half-width to about 3 σ

Practical matters • What about near the edge? – the filter window falls off the edge of the image – need to extrapolate – methods: • • clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner

Practical matters – methods (MATLAB): • • clip filter (black): imfilter(f, g, 0) wrap around: imfilter(f, g, ‘circular’) copy edge: imfilter(f, g, ‘replicate’) reflect across edge: imfilter(f, g, ‘symmetric’) Source: S. Marschner

Q? Practical matters • What is the size of the output? • MATLAB: filter 2(g, f, shape) – shape = ‘full’: output size is sum of sizes of f and g – shape = ‘same’: output size is same as f – shape = ‘valid’: output size is difference of sizes of f and g g full g same g f g valid g g f g g g Source: S. Lazebnik

Application: Representing Texture Source: Forsyth

Texture and Material http: //www-cvr. ai. uiuc. edu/ponce_grp/data/texture_database/samples/

Texture and Orientation http: //www-cvr. ai. uiuc. edu/ponce_grp/data/texture_database/samples/

Texture and Scale http: //www-cvr. ai. uiuc. edu/ponce_grp/data/texture_database/samples/

What is texture? Regular or stochastic patterns caused by bumps, grooves, and/or markings

How can we represent texture? • Compute responses of blobs and edges at various orientations and scales

Overcomplete representation: filter banks LM Filter Bank Code for filter banks: www. robots. ox. ac. uk/~vgg/research/texclass/filters. html

Filter banks • Process image with each filter and keep responses (or squared/abs responses)

How can we represent texture? • Measure responses of blobs and edges at various orientations and scales • Record simple statistics (e. g. , mean, std. ) of absolute filter responses

Can you match the texture to the response? Filters A B 1 2 C 3 Mean abs responses

Representing texture by mean abs response Filters Mean abs responses

Project 1: Hybrid Images Gaussian Filter! A. Oliva, A. Torralba, P. G. Schyns, “Hybrid Images, ” SIGGRAPH 2006 Laplacian Filter! Project Instructions: unit impulse Gaussian Laplacian of Gaussian http: //courses. engr. illinois. edu/cs 498 dh 3/projects/hybrid/Computational. Photography_Project. Hybrid. html

Take-home messages • Image is a matrix of numbers • Linear filtering is a dot product at each position – Can smooth, sharpen, translate (among many other uses) • Be aware of details for filter size, extrapolation, cropping • Start thinking about project (read the paper, create a test project page) = 0. 92 0. 93 0. 94 0. 97 0. 62 0. 37 0. 85 0. 97 0. 93 0. 92 0. 99 0. 95 0. 89 0. 82 0. 89 0. 56 0. 31 0. 75 0. 92 0. 81 0. 95 0. 91 0. 89 0. 72 0. 51 0. 55 0. 51 0. 42 0. 57 0. 41 0. 49 0. 91 0. 92 0. 96 0. 95 0. 88 0. 94 0. 56 0. 46 0. 91 0. 87 0. 90 0. 97 0. 95 0. 71 0. 81 0. 87 0. 57 0. 37 0. 80 0. 88 0. 89 0. 79 0. 85 0. 49 0. 62 0. 60 0. 58 0. 50 0. 61 0. 45 0. 33 0. 86 0. 84 0. 74 0. 58 0. 51 0. 39 0. 73 0. 92 0. 91 0. 49 0. 74 0. 96 0. 67 0. 54 0. 85 0. 48 0. 37 0. 88 0. 90 0. 94 0. 82 0. 93 0. 69 0. 49 0. 56 0. 66 0. 43 0. 42 0. 77 0. 73 0. 71 0. 90 0. 99 0. 73 0. 90 0. 67 0. 33 0. 61 0. 69 0. 73 0. 97 0. 91 0. 94 0. 89 0. 41 0. 78 0. 77 0. 89 0. 93 1 1 1 1 1

Take-home questions 1. Write down a 3 x 3 filter that returns a positive value if the average value of the 4 -adjacent neighbors is less than the center and a negative value otherwise 2. Write down a filter that will compute the gradient in the x-direction: gradx(y, x) = im(y, x+1)-im(y, x) for each x, y

Take-home questions 3. Fill in the blanks: a) b) c) d) _ A F _ Filtering Operator = = D _ D D * * B _ _ D A B E G C F H I D

Next class: Thinking in Frequency