Laplace Approximations Choi Sungjoon DEPARTMENT OF ELECTRICAL ENGINEERING

Laplace Approximations Choi Sungjoon DEPARTMENT OF ELECTRICAL ENGINEERING SEOUL NATIONAL

Contents CPSLAB (cpslab. snu. ac. kr) • Laplace Method – Wikipedia [1774] • Laplace Approximation for posterior mean – Accurate approximations for posterior moments and marginal densities [1986] • Laplace Approximation for non-positive functions – Fully Exponential Laplace Approximations to Expectations and Variances of Non-positive functions [1989] • Example and Implementation Issue – Laplace’s Method Approximations for Probabilistic Inference in Belief Networks with Continuous Variables [1994] 2/23/2021 CPSLAB (EECS, SNU) 2

Caution CPSLAB (cpslab. snu. ac. kr) • Be prepared for enormous equations… 2/23/2021 CPSLAB (EECS, SNU) 3

Contents CPSLAB (cpslab. snu. ac. kr) • Laplace Method – Wikipedia [1774] • Laplace Approximation for posterior mean – Accurate approximations for posterior moments and marginal densities [1986] • Laplace Approximation for non-positive functions – Fully Exponential Laplace Approximations to Expectations and Variances of Non-positive functions [1989] • Example and Implementation Issue – Laplace’s Method Approximations for Probabilistic Inference in Belief Networks with Continuous Variables [1994] 2/23/2021 CPSLAB (EECS, SNU) 4

Laplace Method CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 5

Laplace Method CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 6

Laplace Method CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 7

Laplace Method – example 1 CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 8

Laplace Method – example 1 CPSLAB (cpslab. snu. ac. kr) • MATLAB Source • • • • • • • • • rates = []; Ms = []; for Mval = 0. 1: 100 %% Set function to integrate M = Mval; f = @(x) -x. ^4; fun = @(x) exp(M. *f(x)); %% Numerical int_result = integral( fun, -Inf, Inf); int_result %% Laplace Method x 0 = 0; ff = -12; % funÀÇ second derivative. app_result = exp(M*f(x 0))*sqrt(pi*2/M/abs(ff)); app_result err = (int_result-app_result)/int_result; rate = 1 -abs(err); rates = [rates rate]; Ms = [Ms M]; end %% plot(Ms, rates); title('Error of Laplce Approximation'); xlabel('M'); ylabel('Error'); %% ezplot(fun) 2/23/2021 CPSLAB (EECS, SNU) 9

Laplace Method – example 2 CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 10

Any Questions about Laplace Method? CPSLAB (cpslab. snu. ac. kr) 2/23/2021 CPSLAB (EECS, SNU) 11

Contents CPSLAB (cpslab. snu. ac. kr) • Laplace Method – Wikipedia [1774] • Laplace Approximation for posterior mean – Accurate approximations for posterior moments and marginal densities [1986] • Laplace Approximation for non-positive functions – Fully Exponential Laplace Approximations to Expectations and Variances of Non-positive functions [1989] • Example and Implementation Issue – Laplace’s Method Approximations for Probabilistic Inference in Belief Networks with Continuous Variables [1994] 2/23/2021 CPSLAB (EECS, SNU) 12

What is posterior mean? CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 13

Minimum mean square error(MMSE) estimator CPSLAB (cpslab. snu. ac. kr) 2/23/2021 CPSLAB (EECS, SNU) 14

Approximating Posterior Mean using Laplace Approximation (1/2) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 15

Approximating Posterior Mean using Laplace Approximation (2/2) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 16

Approximating Posterior Variance using Laplace Approximation CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 17

Applying Laplace Approximation CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 18

Applying Laplace Approximation CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 19

Any Questions about Laplace Approximation? CPSLAB (cpslab. snu. ac. kr) 2/23/2021 CPSLAB (EECS, SNU) 20

Contents CPSLAB (cpslab. snu. ac. kr) • Laplace Method – Wikipedia [1774] • Laplace Approximation for posterior mean – Accurate approximations for posterior moments and marginal densities [1986] • Laplace Approximation for non-positive functions – Fully Exponential Laplace Approximations to Expectations and Variances of Non-positive functions [1989] • Example and Implementation Issue – Laplace’s Method Approximations for Probabilistic Inference in Belief Networks with Continuous Variables [1994] 2/23/2021 CPSLAB (EECS, SNU) 21

Laplace Approximation for non-positive functions CPSLAB (cpslab. snu. ac. kr) • n 2/23/2021 CPSLAB (EECS, SNU) 22

Laplace Approximation for non-positive functions CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 23

Basic Laplace approximation to integrals CPSLAB (cpslab. snu. ac. kr) • We can directly (2. 3) into posterior mean formulation. 2/23/2021 CPSLAB (EECS, SNU) 24

CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 25

Moment-generating function (MGF) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 26

Approximation Using the MGF (1/3) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 27

Approximation Using the MGF (2/3) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 28

Approximation Using the MGF (2/3) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 29

Any Questions about Laplace Approximation using moment-generating function? CPSLAB (cpslab. snu. ac. kr) 2/23/2021 CPSLAB (EECS, SNU) 30

Contents CPSLAB (cpslab. snu. ac. kr) • Laplace Method – Wikipedia [1774] • Laplace Approximation for posterior mean – Accurate approximations for posterior moments and marginal densities [1986] • Laplace Approximation for non-positive functions – Fully Exponential Laplace Approximations to Expectations and Variances of Non-positive functions [1989] • Example and Implementation Issue – Laplace’s Method Approximations for Probabilistic Inference in Belief Networks with Continuous Variables [1994] 2/23/2021 CPSLAB (EECS, SNU) 31

Example 1 - Belief net CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 32

Example 2 – Beta distribution (1/3) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 33

Example 2 – Beta distribution (2/3) CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 34

Example 2 – Beta distribution (3/3) CPSLAB (cpslab. snu. ac. kr) 2/23/2021 CPSLAB (EECS, SNU) 35

Monte Carlo Method CPSLAB (cpslab. snu. ac. kr) • 2/23/2021 CPSLAB (EECS, SNU) 36

Q&A CPSLAB (cpslab. snu. ac. kr) Thanks to Jungchan 37