Some Surprises in the Theory of Generalized Belief

Some Surprises in the Theory of Generalized Belief Propagation Jonathan Yedidia Mitsubishi Electric Research Labs (MERL) Collaborators: Bill Freeman (MIT) Yair Weiss (Hebrew University) See “Constructing Free Energy Approximations and Generalized Belief Propagation Algorithms, ” MERL TR 2004 -040, to be published in IEEE Trans. Info. Theory.

Outline • Introduction to GBP – Quick Review of Standard BP – Free Energies – Region Graphs and Valid Approximations • Some Surprises • Maxent Normal Approximations & a useful heuristic

Factor Graphs 1 2 A 3 B (Kschischang, et. al. 2001) 4 C

Computing Marginal Probabilities Fundamental for • Decoding error-correcting codes • Inference in Bayesian networks • Computer vision • Statistical physics of magnets Non-trivial because of the huge number of terms in the sum.

Error-correcting Codes + + (Tanner, 1981 Gallager, 1963) + Marginal Probabilities = A posteriori bit probabilities

Statistical Physics Marginal Probabilities= local magnetization

Standard Belief Propagation i “beliefs” “messages” a The “belief” is the BP approximation of the marginal probability.

BP Message-update Rules we get Using i a = i a

Variational (Gibbs) Free Energy Kullback-Leibler Distance: “Boltzmann’s Law” (serves to define the “energy”): Variational Free Energy is minimized when

So we’ve replaced the intractable problem of computing marginal probabilities with the even more intractable problem of minimizing a variational free energy over the space of possible beliefs. But the point is that now we can introduce interesting approximations.

Region-based Approximations to the Variational Free Energy (Kikuchi, 1951) Exact: Regions: (intractable)

Defining a “Region” A region r is a set of variable nodes Vr and factor nodes Fr such that if a factor node a belongs to Fr , all variable nodes neighboring a must belong to Vr. Regions Not a Region

Region Definitions Region states: Region beliefs: Region energy: Region average energy: Region entropy: Region free energy:

“Valid” Approximations Introduce a set of regions R, and a counting number cr for each region r in R, such that cr=1 for the largest regions, and for every factor node a and variable node i, Indicator functions Count every node once!

Entropy and Energy • : Counting each factor node once makes the approximate energy exact (if the beliefs are). • : Counting each variable node once makes the approximate entropy reasonable (at least the entropy is correct when all states are equiprobable).

Comments • We could actually use different counting numbers for energy and entropy—leads to “fractional BP” (Weigerinck & Heskes 2002) or “convexified free energy” (Wainwright et. al. , 2002) algorithms. • Of course, we also need to impose normalization and consistency constraints on our free energy.

Methods to Generate Valid Region-based Approximations Region Graphs Cluster Variational Method (Kikuchi) Junction graphs Aji-Mc. Eliece Junction trees Bethe (Bethe is example of Kikuchi for Factor graphs with no 4 -cycles; Bethe is example of Aji-Mc. Eliece for “normal” factor graphs. )

Example of a Region Graph A, C, 1, 2, 4, 5 2, 5 B, D, 2, 3, 5, 6 C, E, 4, 5, 7, 8 D, F, 5, 6, 8, 9 C, 4, 5 D, 5, 6 5, 8 [5] is a child of [2, 5] 5

Definition of a Region Graph • Labeled, directed graph of regions. • Arc may exist from region A to region B if B is a subset of A. • Sub-graphs formed from regions containing a given node are connected. • where is the set of ancestors of region r. (Mobius Function) • We insist that

Bethe Method (after Bethe, 1935) 1 Two sets of regions: Large regions containing a single factor node a and all attached variable nodes. 2 A 4 Small regions containing a single variable node i. C 1 B; 2, 3, 5, 6 2 3 D; 5, 6 C; 4, 5 4 B 5 D E 7 A; 1, 2, 4, 5 3 5 6 6 F 8 9 F; 5, 6, 8, 9 E; 4, 5, 7, 8 7 8 9

Bethe Approximation to Gibbs Free Energy Equal to the exact Gibbs free energy when the factor graph is a tree because in that case,

Minimizing the Bethe Free Energy

Bethe = BP Identify to obtain BP equations: i

Cluster Variation Method (Kikuchi, 1951) Form a region graph with an arbitrary number of different sized regions. Start with largest regions. Then find intersection regions of the largest regions, discarding any regions that are sub-regions of other intersection regions. Continue finding intersections of those intersection regions, etc. All intersection regions obey S(r ) is the set of super-regions of region r. , where

Region Graph Created Using CVM A, C, 1, 2, 4, 5 2, 5 B, D, 2, 3, 5, 6 C, E, 4, 5, 7, 8 D, F, 5, 6, 8, 9 C, 4, 5 D, 5, 6 5, 8 5

Minimizing a Region Graph Free Energy • Minimization is possible, but it may be awkward because of all the constraints that must be satisfied. • We introduce generalized belief propagation algorithms whose fixed points are provably identical to the stationary points of the region graph free energy.

Generalized Belief Propagation • Belief in a region is the product of: – Local information (factors in region) – Messages from parent regions – Messages into descendant regions from parents who are not descendants. • Message-update rules obtained by enforcing marginalization constraints.

Generalized Belief Propagation 1 2 3 4 5 6 7 8 9 1245 2356 4578 5689 25 45 56 58 5

Generalized Belief Propagation 1 2 3 4 5 6 7 8 9 1245 2356 4578 5689 25 45 56 58 5

Generalized Belief Propagation 1 2 3 4 5 6 7 8 9 1245 2356 4578 5689 25 45 56 58 5

Generalized Belief Propagation 1 2 3 4 5 6 7 8 9 1245 2356 4578 5689 25 45 56 58 5

Generalized Belief Propagation Use Marginalization Constraints to Derive Message-Update Rules 1 2 3 4 5 6 7 8 9 = 1 2 3 4 5 6 7 8 9

Generalized Belief Propagation Use Marginalization Constraints to Derive Message-Update Rules 1 2 3 4 5 6 7 8 9 = 1 2 3 4 5 6 7 8 9

Generalized Belief Propagation Use Marginalization Constraints to Derive Message-Update Rules 1 2 3 4 5 6 7 8 9 = 1 2 3 4 5 6 7 8 9

Generalized Belief Propagation Use Marginalization Constraints to Derive Message-Update Rules 1 2 3 4 5 6 7 8 9 = 1 2 3 4 5 6 7 8 9

(Mild) Surprise #1 • Region beliefs (even those given by Bethe/BP) may not be realizable as the marginals of any global belief. 1 a 2 b c 3 is a perfectly acceptable solution to BP, but cannot arise from any

(Minor) Surprise #2 • For some sets of beliefs (sometimes even when the marginal beliefs are the exactly correct ones), the Bethe entropy is negative! Each large region has counting number 1, each small region has counting number -2, so if the beliefs are then the entropy is negative:

(Serious) Surprise #3 • When there are no interactions, the minimum of the CVM free energy should correspond to the equiprobable global distribution, but sometimes it doesn’t!

Example Fully connected pairwise model, using CVM and all triplets as the largest regions, with pairs and singlets as the intersection regions. # of regions Triplets Pairs Singlets Counting #

Two simple distributions • Equiprobable distribution: each state is equally probable for all beliefs. Each region gives – e. g. • Distribution obtained by marginalizing the global distribution that only allows all-zeros or all-ones. “Equipolarized distribution” – e. g. Each region gives

Surprise #3 (cont. ) • Surprisingly, the “equipolarized” distribution can have a greater entropy than the equiprobable distribution for some valid CVM approximations. • This is a serious problem, because if the model gets the wrong answer without any interactions, it can’t be expected to be correct with interactions.

The Fix: Consider only Maxent-Normal Approximations • A maxent-normal approximation is one which gives a maximum of the entropy when the beliefs correspond to the equiprobable distribution. • Bethe approximations are provably maxent-normal. • Some other CVM and other region-graph approximations are also provably maxent-normal. – Using quartets on square lattices – Empirically, these approximations give good results • Other valid CVM or region-graph approximations are not maxent-normal. – Empirically, these approximations give poor results

A Very Simple Heuristic • Sum all the counting numbers. – If the sum is greater than N, the equipolarized distribution will have a greater entropy than the equiprobable distribution. (Too Much—Very Bad) – If the sum is less than 0, the equipolarized distribution will have a negative entropy. (Too Little) – If the sum equals 1, the equipolarized distribution will have the correct entropy. (Just Right)

10 x 10 Ising Spin Glass Random fields Random interactions

Conclusions • Standard BP essentially equivalent to minimizing the Bethe free energy. • Bethe method and cluster variation method are special cases of the more general region graph method for generating valid free energy approximations. • GBP is essentially equivalent to minimizing region graph free energy. • One should be careful to use maxent-normal approximations. • Sometimes you can prove that your approximation is maxent-normal; and simple heuristics can be used to prove when it isn’t.