Physical Fluctuomatics 7 th10 th Belief propagation Appendix
Physical Fluctuomatics 7 th~10 th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip. is. tohoku. ac. jp http: //www. smapip. is. tohoku. ac. jp/~kazu/ Physics Fluctuomatics (Tohoku University) 1
Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co. , Ltd. , 2006 (in Japanese) , Chapter 5. References H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011. M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010. Physics Fluctuomatics (Tohoku University) 2
Probabilistic Model for Ferromagnetic Materials Physics Fluctuomatics (Tohoku University) 3
Probabilistic Model for Ferromagnetic Materials > = > Prior probability prefers to the configuration with the least number of red lines. Physics Fluctuomatics (Tohoku University) 4
More is different in Probabilistic Model for Ferromagnetic Materials Sampling by Markov Chain Monte Carlo method Small p Large p Disordered State Ordered State More is different. Critical Point (Large fluctuation) Physics Fluctuomatics (Tohoku University) 5
Fundamental Probabilistic Models for Magnetic Materials +1 -1 Since h is positive, the probablity of up spin is larger than the one of down spin. h :External Field Average Variance Physics Fluctuomatics (Tohoku University) 6
Fundamental Probabilistic Models for Magnetic Materials J :Interaction -1 -1 -1 +1 +1 +1 -1 +1 Since J is positive, (a 1, a 2)=(+1, +1) and ( -1, -1) have the largest probability. Average Variance Physics Fluctuomatics (Tohoku University) 7
Fundamental Probabilistic Models for Magnetic Materials E:Set of All the neighbouring Pairs of Nodes h J Translational Symmetry Problem: Compute Physics Fluctuomatics (Tohoku University) 8
Fundamental Probabilistic Models for Magnetic Materials h J Translational Symmetry Problem: Compute Spontaneous Magnetization Physics Fluctuomatics (Tohoku University) 9
Mean Field Approximation for Ising Model We assume that the probability for configurations satisfying are large. h Jm Jm Physics Fluctuomatics (Tohoku University) i Jm Jm 10
Mean Field Approximation for Ising Model We assume that all random variables ai are independent of each other, approximately. Fixed Point Equation of m Physics Fluctuomatics (Tohoku University) 11
Fixed Point Equation and Iterative Method • Fixed Point Equation Physics Fluctuomatics (Tohoku University) 12
Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University) 13
Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University) 14
Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University) 15
Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University) 16
Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University) 17
Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University) 18
Marginal Probability Distribution in Mean Field Approximation h Jm Jm i Jm Jm Jm:Mean Field Physics Fluctuomatics (Tohoku University) 19
Advanced Mean Field Method Bethe Approximation h l:Effective Field l h J l l h Fixed Point Equation for l Kikuchi Method (Cluster Variation Meth) Physics Fluctuomatics (Tohoku University) 20
Average of Ising Model on Square Grid Graph h J (a) (b) (c) (d) Mean Field Approximation Bethe Approximation Kikuchi Method (Cluster Variation Method) Exact Solution (L. Onsager,C. N. Yang) Physics Fluctuomatics (Tohoku University) 21
Model Representation in Statistical Physics Gibbs Distribution Energy Function Partition Function Free Energy Physics Fluctuomatics (Tohoku University) 22
Gibbs Distribution and Free Energy Gibbs Distribution Free Energy Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a) Free Energy Functional of Trial Probability Distribution Q(a) Physics Fluctuomatics (Tohoku University) 23
Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional Normalization Condition Physics Fluctuomatics (Tohoku University) 24
Kullback-Leibler Divergence and Free Energy Physics Fluctuomatics (Tohoku University) 25
Interpretation of Mean Field Approximation as Information Theory Minimization of Kullback-Leibler Divergence between and Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P] Physics Fluctuomatics (Tohoku University) 26
Interpretation of Mean Field Approximation as Information Theory h J Translational Symmetry Problem: Compute Magnetization Physics Fluctuomatics (Tohoku University) 27
Kullback-Leibler Divergence in Mean Field Approximation for Ising Model Physics Fluctuomatics (Tohoku University) 28
Minimization of Kullback-Leibler Divergence and Mean Field Equation Set of all the neighbouring nodes of the node i Variation i Fixed Point Equations for {Qi| i V} Physics Fluctuomatics (Tohoku University) 29
Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model Physics Fluctuomatics (Tohoku University) 30
Conventional Mean Field Equation in Ising Model h J Translational Symmetry Fixed Point Equation Physics Fluctuomatics (Tohoku University) 31
Interpretation of Bethe Approximation (1) Translational Symmetry h h J J Compute and Physics Fluctuomatics (Tohoku University) 32
Interpretation of Bethe Approximation (2) KL Divergence Free Energy Physics Fluctuomatics (Tohoku University) 33
Interpretation of Bethe Approximation (3) KL Divergence Free Energy Bethe Free Energy Physics Fluctuomatics (Tohoku University) 34
Interpretation of Bethe Approximation (4) Physics Fluctuomatics (Tohoku University) 35
Interpretation of Bethe Approximation (5) Lagrange Multipliers to ensure the constraints Physics Fluctuomatics (Tohoku University) 36
Interpretation of Bethe Approximation (6) • Extremum Condition Physics Fluctuomatics (Tohoku University) 37
Interpretation of Bethe Approximation (7) Extremum Condition Physics Fluctuomatics (Tohoku University) 38
Interpretation of Bethe Approximation (8) Extremum Condition 3 4 1 5 2 4 3 8 1 2 5 6 Physics Fluctuomatics (Tohoku University) 7 39
Interpretation of Bethe Approximation (9) Message Update Rule 3 4 1 5 2 4 3 8 1 2 5 6 Physics Fluctuomatics (Tohoku University) 7 40
Interpretation of Bethe Approximation (10) Message Passing Rule of Belief Propagation 4 3 4 1 2 3 8 1 2 5 6 7 3 5 It corresponds to Bethe approximation in the statistical mechanics. = 4 Physics Fluctuomatics (Tohoku University) 1 2 5 41
Interpretation of Bethe Approximation (11) Translational Symmetry Physics Fluctuomatics (Tohoku University) 42
Summary Statistical Physics and Information Theory Probabilistic Model of Ferromagnetism Mean Field Theory Gibbs Distribution and Free Energy and Kullback-Leibler Divergence Interpretation of Mean Field Approximation as Information Theory Interpretation of Bethe Approximation as Information Theory Physics Fluctuomatics (Tohoku University) 43
- Slides: 43