Outline Texture modeling continued Julesz ensemble 1122020 Visual

Outline • Texture modeling - continued – Julesz ensemble 11/2/2020 Visual Perception Modeling 1

FRAME Model – review • FRAME model – Filtering, random field, and maximum entropy – A well-defined mathematical model for textures by combining filtering and random field models – Maximum entropy is used when constructing the probability distribution on the image space – Minimum entropy is used when selecting filters from a large bank of filters – Together this is called min-max entropy principle 11/2/2020 Visual Perception Modeling 2

FRAME Model – review • Maximum Entropy Distribution – Given the expectations of some functions, the maximum entropy solution for p(x) is – where 11/2/2020 Visual Perception Modeling 3

FRAME Model – review • Maximum Entropy – continued – are determined by the constraints – Gradient ascend to maximize 11/2/2020 Visual Perception Modeling 4

Julesz Ensemble • The original texture modeling question – What features and statistics are characteristics of a texture pattern, so that texture pairs that share the same features and statistics cannot be told apart by pre-attentive human visual perception? --- Julesz, 1962 11/2/2020 Visual Perception Modeling 5

Summary of Existing Texture Features 11/2/2020 Visual Perception Modeling 6

Existing Feature Statistics 11/2/2020 Visual Perception Modeling 7

Most General Feature Statistics 11/2/2020 Visual Perception Modeling 8

Julesz Ensemble – cont. • Definition – Given a set of normalized statistics on lattice a Julesz ensemble W(h) is the limit of the following set as Z 2 and H {h} under some boundary conditions 11/2/2020 Visual Perception Modeling 9

Julesz Ensemble – cont. • Feature selection – A feature can be selected from a large set of features through information gain, or the decrease in entropy 11/2/2020 Visual Perception Modeling 10

Julesz Ensemble – cont. 11/2/2020 Visual Perception Modeling 11

Julesz Ensemble – cont. • Sampling the Julesz ensemble – In the Julesz ensemble, a texture type is defined as all the images sharing the observed statistics and features • It is an inverse problem in order to generate texture images or verify the statistics – The problem is again the dimensionality • If the image size is 256 x 256 and each pixel can have 8 values, there are 865536 different images – Markov chain Monte-Carlo algorithms 11/2/2020 Visual Perception Modeling 12

Julesz Ensemble – cont. • Given observed feature statistics {H(a)obs}, we associate an energy with any image I as • Then the corresponding Gibbs distribution is – The q(I) can be sampled using a Gibbs sampler or other Markov chain Monte-Carlo algorithms 11/2/2020 Visual Perception Modeling 13

Image Synthesis Algorithm • Compute {Hobs} from an observed texture image • Initialize Isyn as any image, and T as T 0 • Repeat Randomly pick a pixel v in Isyn Calculate the conditional probability q(Isyn(v)| Isyn(-v)) Choose new Isyn(v) under q(Isyn(v)| Isyn(-v)) Reduce T gradually • Until E(I) < e 11/2/2020 Visual Perception Modeling 14

A Texture Synthesis Example Observed image 11/2/2020 Initial synthesized image Visual Perception Modeling 15

A Texture Synthesis Example Temperature 11/2/2020 Image patch Energy Visual Perception Modeling Conditional probability 16

A Texture Synthesis Example - continued Average spectral histogram error 11/2/2020 Visual Perception Modeling 17

Texture Synthesis Examples - continued Observed image 11/2/2020 Synthesized image Visual Perception Modeling 18

Texture Synthesis Examples - continued Observed image 11/2/2020 Synthesized image Visual Perception Modeling 19

Texture Synthesis Examples - continued Mud image Synthesized image 11/2/2020 Visual Perception Modeling 20

Texture Synthesis Examples - continued Observed image Synthesized image 11/2/2020 Visual Perception Modeling 21

Texture Synthesis Examples - continued Observed image 11/2/2020 Synthesized image Visual Perception Modeling 22

Texture Synthesis Examples - continued Original cheetah skin patch 11/2/2020 Visual Perception Modeling Synthesized image 23

Texture Synthesis Examples - continued Observed image 11/2/2020 Synthesized image Visual Perception Modeling 24

Texture Synthesis Examples - continued Observed image 11/2/2020 Synthesized image Visual Perception Modeling 25

Texture Synthesis Examples - continued Observed image 11/2/2020 Synthesized image Visual Perception Modeling 26

An Synthesis Example for Fun 11/2/2020 Visual Perception Modeling 27

Comparison with Texture Synthesis Method - continued • An example from Heeger and Bergen’s algorithm Cross image 11/2/2020 Heeger and Bergen’s Visual Perception Modeling Our result 28

Julesz Ensemble – cont. • Remarks – The results shown here are based on histograms of filter responses – However, the Julesz ensemble applies to any features/statistics of your choice – You can also define Julesz ensemble for images other than textures 11/2/2020 Visual Perception Modeling 29

Julesz Ensemble – cont. • Applications – This essentially provides a framework to systematically verify the sufficiency of chosen features/statistics • Normally, features/statistics are evaluated empirically. In other words, features are evaluated on a limited number of images 11/2/2020 Visual Perception Modeling 30