Embedding Gestalt Laws in Markov Random Fields by

Embedding Gestalt Laws in Markov Random Fields by Song-Chun Zhu

Purpose of the Paper Proposes functions to measure Gestalt features of shapes Adapts [Zhu, Wu Mumford] FRAME method to shapes Exhibits effect of MRF model obtained by putting these together.

Recall Gestalt Features (à la [Lowe], and others) Colinearity Cocircularity Proximity Parallelism Symmetry Continuity Closure Familiarity

FRAME [Zhu, Wu, Mumford] F ilters R andom fields A nd M aximum E ntropy A general procedure for constructing MRF models

Three Main Parts Data Learn MRF models from data Test generative power of learned model

Elements of Data A set of images representative of the chosen application domain An adequate collection of feature measures or filters The (marginal) statistics of applying the feature measures or filters to the set of images

Data: Images Zhu considers 22 animal shapes and their horizontal flips The resulting histograms are symmetric More data can be obtained But are there other effects?

Sample Animate Images

Contour-based Feature Measures Goal is to be generic But generic shape features are hard to find φ1 = κ(s), the curvature κ(s) = 0 implies the linelets on either side of Γ(s) are colinear φ2 = κ'(s), its derivative κ'(s) = 0 implies three sequential linelets are cocircular “Other contour-based shape filters can be defined in the same way”

Zhu's Symmetry Function Ψ(s) pairs linelets across medial axes Defined and computed by minimizing an energy functional constructed so that Paired linelets are as close, parallel and symmetric as possible, and There as few discontinuities as possible

Region-based Feature Measures φ3(s) = dist(s, ψ(s)) Measures proximity of paired linelets across a region φ4(s) = φ3'(s), the derivative φ4(s) = 0 implies paired linelets are parallel φ5(s) = φ'4(s) = φ3''(s) φ5(s) = 0 implies paired linelets are symmetric

Another Possible Shape Feature φ6(s) = 1 where ψ(s) is discontinuous 0 otherwise Counts the number of “parts” a shape has Can Gestalt “familiarity” be (statistically? ) measured?

The Statistic The histogram of feature φ over curve Γ is H(z; φk, Γ) = ∫δ(z-φk(s)) ds δ is the Dirac function: mass 1 at 0, and 0 otherwise μ(z; φk) denotes the average over all images Zhu claims μ is a close estimation of the marginal distribution of the “true distribution” over shape space, assuming the total number of linelets is small.

Statistical Observations On 22 images and their flips φ1 at scales 0, 1, 2 φ3 φ4 φ5

Construct a Model Ω is the space of shapes Φ is a finite subset of feature filters We seek a probability distribution p on Ω ∫Ω p(Γ) dΓ = 1 (1) That reproduces the statistics for all φ in Ω ∫Ω p(Γ) δ(z-φ(s)) dΓ = μ(z; φ) (2)

Construct a Model, 2 Idea: Choose the p with maximal entropy Seems reasonable and fair, but is it really the best target/energy function? Lagrange multipliers and calculus of variations lead to p(Γ; Φ, Λ) = exp(–∑φЄΦ ∫ λ (z) H(φ, Γ, z) dz) / Z φ where Z is the usual normalizing factor Λ = { λ φ | φЄΦ }

It's a Gibbs Distribution In other words, it has the form of a Gibbs distribution, and therefore determines a Markov Random Field (MRF) model.

Markov Chain Monte Carlo Too hard to compute λ's and p analytically Idea: Sample Ω according to the distribution p, stochastically update Λ to update p, and repeat until p reproduces all μ(z; φ) for φ Є Φ Monte Carlo because of random walk Markov Chain in the nature of the loop

Markov Chain Monte Carlo, 2 From the sampling produce μ'(z; φ) Same as μ(z; φ) except based on a random sample of shape space For the purposes of today's discussion, the details are not important For φ Є Φ μ'(z; φ) = μ(z; φ) Zhu et al. assume there exists a “true underlying distribution”

The Nonaccidental Statistic For φ' not in the set Φ we expect μ'(z; φ') ≠ μ(z; φ') μ'(z; φ') is the accidental statistic for φ' It is a measure of correlation between φ' and Φ The “distance” (L 1, L 2, or other) between μ'(z; φ') and μ(z; φ') is the nonaccidental statistic for φ' It is a measure of how much “additional information” φ' carries above what is already in Φ

The Algorithm (simplified) Enter your set Γ = { γ } of shapes Enter a (large) set { φ } of candidate feature measures Compute μ(φ, Γ) for all φ in Φ Compute μ'(φ) relative to a uniform distribution on Ω Until the nonaccidental statistic of all unused features is small enough, repeat:

Algorithm, 2 Of the remaining φ , add to Φ one with maximal nonaccidental statistic Update: Set of Lagrange multipliers Λ = { λ } Probability distribution model p(Φ, Λ) The μ'(φ) for remaining candidate features φ

Experiments and Discussion Let my description of these experiments stimulate your thoughts on such issues as Are there better Gestalt feature measures? What is the best possible outcome of a generative model of shape? What feature measures should be added to the Gestalt ones? How useful were these experiments and what other might be worth doing?

Experiment 1 When the only feature used is the curvature the model generated

Experiment 1, continued A Gaussian model (with the same κ-variance) produced

Experiment 2 uses both κ and κ' The nonaccidental statistic of κ' with respect to the model based on κ can be seen here

Experiment 2, continued This time the model generated these shapes, purported to be smoother and more scale invariant

Experiment 3 The nonaccidental statistics of the three regionbased shape features relative to the model produced in Experiment 2

Experiment 3, continued So r'' was omitted, this model has Φ = { κ, κ', r, r' }

Experiment 3, continued This model produced such shapes as

Concluding Discussion Zhu acknowledges that the selection of training shapes might introduce a bias; but

Discussion, continued Zhu acknowledges that the paucity of Gestalt features limits the possible neighborhood structures used to define a MRF. Zhu acknowledges that these models do not account for high-level shape properties, and suggests that a composition system might address this problem.

Questions and Comments Although it is in the nature of an MRF-model to propagate local properties, I think there needs to be a higher-level basis (than linelets) for measuring the Gestalt features of a shape! Are there better Gestalt feature measures? What feature measures should be added to the Gestalt ones?

More Questions for Discussion What is the best possible outcome of a generative model of shape? Is such a thing worth pursuing? How useful were Zhu' experiments and what others might be worth doing?