Background Probabilistic information processing Inference for Bayesian networks
Background Probabilistic information processing • Inference for Bayesian networks • Error correcting codes 000101 noise decode 000111 000101 • CDMA multi-user detection • Probabilistic image processing degrade image restored image Task High-dimensional distribution Marginals Intractable
Task Intractable • Belief propagation (Sum product algorithm) Bethe free energy Optimization techniques • BO (M. Welling, et. al. , 2001) • CCCP (A. L. Yuille, 2002) • SEQ (Tonosaki, et. al. , 2007) (Approximate) marginals
Performance of CCCP • guarantees to monotonically decrease Bethe free energy unlike belief propagation. • requires huge computational cost compared to belief propagation. • does not always converge for synchronous update of inner loop.
Purpose We extend CCCP algorithm for Bethe free energy and present a new CCCP (NCCCP) algorithm. New CCCP (NCCCP) algorithm • includes conventional CCCP. • guarantees to monotonically decrease Bethe free energy. • is more stable even for synchronous inner loop. • can reduce huge computational cost underlying CCCP.
Concave and convex procedure(CCCP) CCCP algorithm guarantees to monotonically decrease the function(al) by the update rule Convex Extremum Concave
Bethe free energy Concave Convex Conventional CCCP Approximate marginals
Main result (Key idea) NCCCP Bethe free energy Trivial pair creation an arbitrary convex functional
NCCCP Particularly, CCCP Free parameters
CCCP algorithm NCCCP algorithm Outer loop Inner loop Outer loop Approximate marginals
NCCCP algorithm for Bethe free energy Theorem (Outer loop) Outer loop of NCCCP algorithm for minimizing Bethe free energy is given by as follows: where satisfies for all
NCCCP algorithm for Bethe free energy Theorem (Inner loop) Inner loop of NCCCP is given by as follows:
Update Manner of Inner Loop Asynchronous update time Theorem (Inner loop) Inner loop of NCCCP is given by as follows: time CCCP guarantees to converge. Synchronous update time CCCP does not always converge. NCCCP converges.
Role of free parameters When are small When Outer loop slow or Outer loop are large convergence slow or convergence Approximate marginals Outer loop fast Approximate marginals There exist optimal values .
Numerical Results(1/2) (i) Asynchronous inner loop CCCP
Numerical Results (2/2) (ii) Synchronous inner loop CCCP
Conclusion We presented a new CCCP (NCCCP) algorithm for Bethe free energy. New CCCP (NCCCP) algorithm • includes conventional CCCP. • guarantees to monotonically decrease Bethe free energy. • is more stable even for synchronous inner loop. • can reduce huge computational cost underlying CCCP.
Future works • To design efficient NCCCP algorithm based on the optimality of free parameters. • To apply NCCCP algorithm to practical problems such as CDMA multi-user detection problems or decoding algorithm for LDPC codes. 1. 外崎幸徳,樺島祥介, “CCCPに基づくCDMAマルチユーザ検出アルゴリズム, ” 電子情報通信学会論文誌 D, vol. J 89 -D, no. 5, pp. 1049 -1060, 2006. 2. T. Shibuya, K. Harada, R. Tohyama, and K. Sakaniwa, “Iterative Decoding Based on the Concave-Convex Procedure, ” IEICE Trans. Fundam. Electron. Commu. Comput. Sci. , vol. E 88 -A, no. 5, pp. 1346 -1364, 2005.
Example: Gaussian Distributions NCCCP algorithm= Outer loop + Inner loop Theorem (Outer loop in Gaussian distributions) Theorem (Inner loop in Gaussian distributions)
- Slides: 19