ExpectationPropagation performs smooth gradient descent Advances in Approximate

  • Slides: 21
Download presentation
Expectation-Propagation performs smooth gradient descent Advances in Approximate Bayesian Inference 2016 1 GUILLAUME DEHAENE

Expectation-Propagation performs smooth gradient descent Advances in Approximate Bayesian Inference 2016 1 GUILLAUME DEHAENE

Computational troubles in Bayesia 2 �

Computational troubles in Bayesia 2 �

Laplace + Gradient Descent 3 � Probability

Laplace + Gradient Descent 3 � Probability

Laplace + Gradient Descent 4 �

Laplace + Gradient Descent 4 �

Physical intuitions 5 � - Log probability

Physical intuitions 5 � - Log probability

Linking GD, VB and EP 6 VB and EP iterate Gaussian approximations We can

Linking GD, VB and EP 6 VB and EP iterate Gaussian approximations We can define an algorithm that: - Iterates Gaussian - Computes the Laplace - Does Gradient Descent

Algorithm 1: disguised gradient descent 7 �

Algorithm 1: disguised gradient descent 7 �

Algorithm 1: disguised gradient descent 8 �

Algorithm 1: disguised gradient descent 8 �

Variational Bayes Gaussian approximation 9 �

Variational Bayes Gaussian approximation 9 �

Algorithm 2: smoothed gradient descent 10 �

Algorithm 2: smoothed gradient descent 10 �

Algorithm 2: smoothed gradient descent 11

Algorithm 2: smoothed gradient descent 11

Algorithm 3: hybrid smoothing GD 13 �

Algorithm 3: hybrid smoothing GD 13 �

Interpreting algorithm 3 14 �

Interpreting algorithm 3 14 �

Expectation Propagation 15 �

Expectation Propagation 15 �

Algorithm 4: classic Expectation Propagation 16 �

Algorithm 4: classic Expectation Propagation 16 �

Algorithm 5: smooth EP 17 �

Algorithm 5: smooth EP 17 �

Algorithm 5: smooth EP 18 �

Algorithm 5: smooth EP 18 �

Classic vs Smooth EP 19 Algorithm 4: - Computationally efficient - Completely unintuitive Algorithm

Classic vs Smooth EP 19 Algorithm 4: - Computationally efficient - Completely unintuitive Algorithm 5: - Intuitive: linked to Newton’s method - Tractable to analysis Which should we choose?

Conclusion 20 �

Conclusion 20 �

Conclusion 21 �

Conclusion 21 �

Gradient Methods Steepest Descent gradient descent Conjugate GradientGradient Methods Steepest Descent gradient descent Conjugate Gradient
Gradient Descent Review Gradient Descent In step 3Gradient Descent Review Gradient Descent In step 3
gradient Descent py def gradient Descentx y thetagradient Descent py def gradient Descentx y theta
Training and optimization Stochastic gradient descent Gradient onTraining and optimization Stochastic gradient descent Gradient on
1 Families Kinship and Descent Families Descent Kinship1 Families Kinship and Descent Families Descent Kinship
Recursive descent parsing Some notes on recursive descentRecursive descent parsing Some notes on recursive descent
Single Seed Descent SSD Single seed descent canSingle Seed Descent SSD Single seed descent can
Descent into Savagery Lord of the Flies DescentDescent into Savagery Lord of the Flies Descent
Kinship and Descent Chapter Outline What are descentKinship and Descent Chapter Outline What are descent
Smooth Curves Dr Scott Schaefer 1 Smooth CurvesSmooth Curves Dr Scott Schaefer 1 Smooth Curves
Smooth muscle Smooth Muscle A fusiform fibers BSmooth muscle Smooth Muscle A fusiform fibers B
facebook Wall Photos Me Sources Smooth ER Smoothfacebook Wall Photos Me Sources Smooth ER Smooth
General Shape of a Smooth Muscle Cell SmoothGeneral Shape of a Smooth Muscle Cell Smooth
Smooth Surfaces Dr Scott Schaefer 1 Smooth SurfacesSmooth Surfaces Dr Scott Schaefer 1 Smooth Surfaces
Smooth Muscle Prof K Sivapalan Smooth Muscle FoundSmooth Muscle Prof K Sivapalan Smooth Muscle Found
Excitation and contraction of smooth muscle Smooth muscleExcitation and contraction of smooth muscle Smooth muscle
Chapter 8 Smooth Muscles Smooth Muscles Location StructureChapter 8 Smooth Muscles Smooth Muscles Location Structure
Part 3 Smooth Muscles Smooth muscle cells arePart 3 Smooth Muscles Smooth muscle cells are
Physiology of Smooth Muscle A Structure of SmoothPhysiology of Smooth Muscle A Structure of Smooth
Gradient Descent Slides adapted from David Kauchak MichaelGradient Descent Slides adapted from David Kauchak Michael