Texture ECE 847 Digital Image Processing Stan Birchfield

Texture ECE 847: Digital Image Processing Stan Birchfield Clemson University

What is texture? • No formal definition exists • Basically, repetition of a pattern – Pattern can be deterministic or stochastic – Repetition can be deterministic or stochastic (rep/patt) DD DS SD SS checkerboard, bricks, tiles quilt, mosaic scattered coins grass, pebbles, leopard spots

Another definition Texture is the abstraction of certain homogeneous statistical properties from a portion of the visual field containing a quantity of information in excess of the observer’s perceptual capacity. http: //www. cns. nyu. edu/~msl/papers/landygraham 02. pdf

Uses of texture segmentation S. C. Zhu, PAMI 1996 shape from texture synthesis (texture mapping) http: //graphics. stanford. edu/papers/texture-synthesis-sig 01/

Textons Preattentive texture discrimination (0. 1 -0. 4 sec) Julesz’ psychophysics experiments in the 1960 s: The fundamental elements are textons (elongated blobs, terminators, crossings)

Things vs. Stuff “Things” are often emphasized (object recognition) count nouns Adelson, SPIE 2001 http: //web. mit. edu/persci/people/adelson/pub_pdfs/adelson_spie_01. pdf But “stuff” is important, too (material properties) mass nouns

Statistical representation • Texture is usually represented using statistics of the image, captured at various frequencies and orientations • Frequencies can also be thought of as scales

Co-occurrence matrices • features: http: //www. cse. msu. edu/~stockman/CV/F 09 Lectures/Storehouse/week 06 -texture-LS. ppt

Laws’ masks • 1 D convolution kernels: (Level, Edge, Spot, Wave, Ripple) • • • Combine to 2 D Apply kernels to image Perform windowing operation Normalize for contrast Combine similar features: • Each pixel is now represented by 14 values http: //www. ccs 3. lanl. gov/~kelly/notebook/laws. shtml K. Laws. Textured Image Segmentation, Ph. D. Dissertation, University of Southern California, January 1980.

water tiger fence flag grass small flowers big flowers http: //www. cse. msu. edu/~stockman/CV/F 09 Lectures/Storehouse/week 06 -texture-LS. ppt

Fourier transform

Short-time Fourier transform • Multiply signal by windowing function • Then compute Fourier transform • AKA windowed Fourier transform • Drawback: Fixed resolution forces tradeoff – Either good time resolution or – Good frequency resolution, but not both

Discrete Fourier Transform (DFT) • Signal g(x), x=0, . . . , w-1 – spatial domain • DFT G(k), k=0, . . . , w-1 – frequency domain Forward: (analysis) Inverse: (synthesis) only difference is sign

DFT implementation • Euler’s formula: • Slow version is O(n 2) (where n=w) • Fast version is Fast Fourier Transform (FFT), O(n log n)

Rectangular and polar coordinates } } rectangular polar Gi |G| Gr

DFT properties • Linear • Periodic • Shift theorem • Modulation

DFT properties (cont. ) • Hermitian symmetry (if g is real-valued) • Unitarity (Parseval’s theorem) • DC component • Circular convolution

Orthogonality of basis vectors

Example What is the DFT of this signal?

Example (cont. ) Answer: But why? What does this mean?

Example (cont. ) Other ways of looking at the same data: f = k/w is the frequency (in cycles per sample) -½ ≤ f ≤ ½ The period T=1/f in samples per cycle

More examples What are their DFTs?

More examples (cont. ) Why?

More examples What are their DFTs?

More examples (cont. ) Why?

Multiple sinusoids

DFT as matrix multiplication

2 D DFT • Fwd and inverse 2 D DFT: where

2 D DFT properties • Linear transform of (x, y) • Scaling (special case) • Rotation (special case)

Discrete Fourier Transform (DFT) White pixel is DC component (black rectangle provides contrast) log magnitude = log |G(I(x, y))| phase

DFT (cont. ) slice through DFT (first row): DC component |G(I(x, y))| log |G(I(x, y))|

Periodicity Intention is to take the DFT of the image: DFT

Periodicity DC component But the DFT sees the replicated signal and computes a replicated transform: Strong horizontal edge leads to high coefficients for ej 2 pky/h Shifted DFT is easier to view

Periodicity. . . and carries the same information: A B A B C D C D non-shifted DFT Shifted DFT

Periodicity To shift DFT, first multiply image by (-1)x+y DFT x +1 -1 +1 -1 +1 -1 +1 -1 -1 +1 -1 +1 -1 +1 -1 +1

Discrete wavelet transform • Apply quadrature mirror filter: or g is highpass where (downsample operator) h is lowpass • Successive applications: filter bank

Lifting scheme • Lifting scheme performs DWT by a series of convolution-accumulate operations • Restricted to perfect reconstruction filterbanks • Enables speedup by factor of two

Another wavelet transform Captures frequencies at times

Haar wavelet • A wavelet is described by – mother wavelet function y( (x-b)/a ), and – scaling function j • Haar is oldest (1909) and simplest wavelet

Daughter wavelets • Daughter wavelet functions given by – dilations (change in frequency), and – translations of mother wavelet y( (x-b)/a ) • Usually a=2 -j (scaling by octaves) b = k 2 -j (translation keeps well-separated) • Draw dilated/translated Haar wavelets here

Implementing Haar • Haar is easy to implement: g = [ 1 1 ] (average) h = [1 -1 ] (difference) • Steps: – Convert sequence to pairs – Right-multiply by Haar matrix to yield result • Haar wavelets are complete and orthogonal • Allow complete reconstruction http: //en. wikipedia. org/wiki/Haar_wavelet

2 D Haar

Wavelet packet decomposition • Process both the detail (high-pass) and approximation (low-pass) coefficients

Daubechies wavelets • Daubechies (1987) are popular family of orthogonal wavelets • Characterized by a maximal number of vanishing points for a given support • DN (N is even) has – N coefficients, and – N/2 vanishing moments • Examples of scaling functions: – D 2 wavelet = [1 -1] (Haar) scaling = [1 1 ] – D 4 wavelet = [-0. 1830127, -0. 3169873, 1. 1830127, -0. 6830127] scaling = [0. 6830127, 1. 1830127, 0. 3169873, -0. 1830127] – … D 20 Note: scaling is derived by reversing order of wavelet, and changing sign of every other coefficient

Gabor filter

Gabor pyramid

Biological plausibility • Hubel and Wiesel (1968)

Multiresolution analysis • Mallat (1989) connected wavelets with multiresolution processing • See also image pyramids • Scale space

Steerable pyramids • (Simoncelli 1995) • Outgrowth of Laplacian pyramids • Like DWT, – decomposes image into oriented, bandpass filtered components – uses binary scales • Unlike DWT, – avoids aliasing – shift-invariant – rotation-invariant http: //citeseer. ist. psu. edu/castleman 98 simplified. html

Efros-Leung texture synthesis http: //graphics. cmu. edu/people/efros/research/Efros. Leung. html

Efros-Leung texture synthesis function Grow. Image(Sample. Image, Window. Size) while Image not filled do progress = 0 Pixel. List = Get. Unfilled. Neighbors(Image) foreach Pixel in Pixel. List do Template = Get. Neighborhood. Window(Pixel) Best. Matches = Find. Matches(Template, Sample. Image) Best. Match = Random. Pick(Best. Matches) if (Best. Match. error < Max. Err. Threshold) then Pixel. value = Best. Match. value progress = 1 end if progress == 0 then Max. Err. Threshold = Max. Err. Threshold * 1. 1 end return Image end http: //graphics. cmu. edu/people/efros/research/NPS/alg. html

Efros-Leung texture synthesis function Find. Matches(Template, Sample. Image) Valid. Mask = 1 s where Template is filled, 0 s otherwise Gauss. Mask = Gaussian 2 D(Window. Size, Sigma) Tot. Weight = sum i, j Gaussi. Mask(i, j)*Valid. Mask(i, j) for i, j do for ii, jj do dist = (Template(ii, jj)-Sample. Image(i-ii, j-jj))^2 SSD(i, j) = SSD(i, j) + dist*Valid. Mask(ii, jj)*Gauss. Mask(ii, jj) end SSD(i, j) = SSD(i, j) / Tot. Weight end Pixel. List = all pixels (i, j) where SSD(i, j) <= min(SSD)*(1+Err. Threshold) return Pixel. List end http: //graphics. cmu. edu/people/efros/research/NPS/alg. html

Efros-Leung algorithm For each unfilled pixel that is neighbor to a filled pixel, 1. Select (square) window around pixel 2. Cross-correlate with rest of image (or texture patch) 3. Randomly select a location that yields a good match 4. If error < max_error, then set pixel value 5. Else go to another pixel If no progress after all pixels have been considered, then increment max_error and try again

Random selection How to randomly select location? • Form PDF from correlation values (retain only those within some percentage of peak • Compute running sum (CDF) • Generate random number between 0 and 1 • Find first location whose value in CDF exceeds number This will select location according to probability density function resulting from cross-correlation