Expectation Maximization EM Northwestern University EECS 395495 Special

  • Slides: 37
Download presentation
Expectation Maximization (EM) Northwestern University EECS 395/495 Special Topics in Machine Learning Most slides

Expectation Maximization (EM) Northwestern University EECS 395/495 Special Topics in Machine Learning Most slides from http: //www. autonlab. org/tutorials/

Outline • Objective • Simple example • Complex example Most slides from http: //www.

Outline • Objective • Simple example • Complex example Most slides from http: //www. autonlab. org/tutorials/

Objective • Learning with missing/unobservable data E B A J 1 1 B E

Objective • Learning with missing/unobservable data E B A J 1 1 B E 1 0 1 1 0 0 A … Maximum likelihood J Most slides from http: //www. autonlab. org/tutorials/

Objective • Learning with missing/unobservable data E B A J 1 1 ? 1

Objective • Learning with missing/unobservable data E B A J 1 1 ? 1 B E 1 0 ? 1 0 0 ? 0 … A Optimize what? J Most slides from http: //www. autonlab. org/tutorials/

Outline • Objective • Simple example • Complex example Most slides from http: //www.

Outline • Objective • Simple example • Complex example Most slides from http: //www. autonlab. org/tutorials/

Simple example Most slides from http: //www. autonlab. org/tutorials/

Simple example Most slides from http: //www. autonlab. org/tutorials/

Maximize likelihood Most slides from http: //www. autonlab. org/tutorials/

Maximize likelihood Most slides from http: //www. autonlab. org/tutorials/

Same Problem with Hidden Information Hidden Observable Most slides from http: //www. autonlab. org/tutorials/

Same Problem with Hidden Information Hidden Observable Most slides from http: //www. autonlab. org/tutorials/ Grade Score

Same Problem with Hidden Information G S Most slides from http: //www. autonlab. org/tutorials/

Same Problem with Hidden Information G S Most slides from http: //www. autonlab. org/tutorials/

Same Problem with Hidden Information G S Most slides from http: //www. autonlab. org/tutorials/

Same Problem with Hidden Information G S Most slides from http: //www. autonlab. org/tutorials/

EM for our example Most slides from http: //www. autonlab. org/tutorials/

EM for our example Most slides from http: //www. autonlab. org/tutorials/

EM Convergence Most slides from http: //www. autonlab. org/tutorials/

EM Convergence Most slides from http: //www. autonlab. org/tutorials/

Generalization • X: observable data (score = {h, c, d}) • z: missing data

Generalization • X: observable data (score = {h, c, d}) • z: missing data (grade = {a, b, c, d}) • : model parameters to estimate ( ) • E: given , compute the expectation of z • M: use z obtained in E step, maximize the likelihood with respect to Most slides from http: //www. autonlab. org/tutorials/

Outline • Objective • Simple example • Complex example Most slides from http: //www.

Outline • Objective • Simple example • Complex example Most slides from http: //www. autonlab. org/tutorials/

Gaussian Mixtures Most slides from http: //www. autonlab. org/tutorials/

Gaussian Mixtures Most slides from http: //www. autonlab. org/tutorials/

Gaussian Mixtures • Know – Data – – - • Don’t know – Data

Gaussian Mixtures • Know – Data – – - • Don’t know – Data label • Objective – Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The GMM assumption Most slides from http: //www. autonlab. org/tutorials/

The data generated Label Coordinates Most slides from http: //www. autonlab. org/tutorials/

The data generated Label Coordinates Most slides from http: //www. autonlab. org/tutorials/

Computing the likelihood Most slides from http: //www. autonlab. org/tutorials/

Computing the likelihood Most slides from http: //www. autonlab. org/tutorials/

EM for GMMs Most slides from http: //www. autonlab. org/tutorials/

EM for GMMs Most slides from http: //www. autonlab. org/tutorials/

EM for GMMs Most slides from http: //www. autonlab. org/tutorials/

EM for GMMs Most slides from http: //www. autonlab. org/tutorials/

EM for GMMs Most slides from http: //www. autonlab. org/tutorials/

EM for GMMs Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Most slides from http: //www. autonlab. org/tutorials/

Generalization • X: observable data • z: unobservable data • : model parameters to

Generalization • X: observable data • z: unobservable data • : model parameters to estimate • E: given , compute the “expectation” of z • M: use z obtained in E step, maximize the likelihood with respect to Most slides from http: //www. autonlab. org/tutorials/

For distributions in exponential family • Exponential family – Yes: normal, exponential, beta, Bernoulli,

For distributions in exponential family • Exponential family – Yes: normal, exponential, beta, Bernoulli, binomial, multinomial, Poisson… – No: Cauchy and uniform • EM using sufficient statistics – S 1: computing the expectation of the statistics – S 2: set the maximum likelihood Most slides from http: //www. autonlab. org/tutorials/

What EM really is • X: observable data • Maximize expected log likelihood •

What EM really is • X: observable data • Maximize expected log likelihood • z: missing data • E-step: Determine the expectation • M-step: Maximize the expectation above with respect to Most slides from http: //www. autonlab. org/tutorials/

Final comments • Deal with missing data/latent variables • Maximize expected log likelihood •

Final comments • Deal with missing data/latent variables • Maximize expected log likelihood • Local minima Most slides from http: //www. autonlab. org/tutorials/

EE148 Expectation Maximization Markus Weber 51199 Overview ExpectationEE148 Expectation Maximization Markus Weber 51199 Overview Expectation
Chapter 9 Profit maximization Profit maximization n nChapter 9 Profit maximization Profit maximization n n
Profit Maximization in a Monopoly Profit Maximization inProfit Maximization in a Monopoly Profit Maximization in
Maximization without Calculus Not all economic maximization problemsMaximization without Calculus Not all economic maximization problems
Maximization without Calculus Not all economic maximization problemsMaximization without Calculus Not all economic maximization problems
Profit Maximization Chapter 9 1 Profit Maximization TheProfit Maximization Chapter 9 1 Profit Maximization The
EECS 395495 Algorithmic Techniques for Bioinformatics General IntroductionEECS 395495 Algorithmic Techniques for Bioinformatics General Introduction
Important o EECS 395495 Io T Networks SeminarImportant o EECS 395495 Io T Networks Seminar
Welcome to EECS 395495 Networking Problems in CloudWelcome to EECS 395495 Networking Problems in Cloud
Chapter 11 Northwestern Europe Countries of Northwestern EuropeChapter 11 Northwestern Europe Countries of Northwestern Europe
Advanced Placement Program Northwestern High School Welcome NorthwesternAdvanced Placement Program Northwestern High School Welcome Northwestern
Maximum Likelihood Estimation Expectation Maximization Lectures 3 OctMaximum Likelihood Estimation Expectation Maximization Lectures 3 Oct
Handout 5 Expectation Maximization EM Unsupervised Learning andHandout 5 Expectation Maximization EM Unsupervised Learning and
Expectation Maximization Algorithm Rong Jin A Mixture ModelExpectation Maximization Algorithm Rong Jin A Mixture Model
Pat Reco Expectation Maximization Alexandros Potamianos School ofPat Reco Expectation Maximization Alexandros Potamianos School of
CS 2750 Machine Learning Expectation Maximization Prof AdrianaCS 2750 Machine Learning Expectation Maximization Prof Adriana
Gaussian Mixture Models and Expectation Maximization The ProblemGaussian Mixture Models and Expectation Maximization The Problem
An Introduction to the Expectation Maximization EM AlgorithmAn Introduction to the Expectation Maximization EM Algorithm
Expectation Maximization for GMM Comp 344 Tutorial KaiExpectation Maximization for GMM Comp 344 Tutorial Kai
The Expectation Maximization EM Algorithm continued 600 465The Expectation Maximization EM Algorithm continued 600 465
Expectation Maximization First introduced in 1977 Lots ofExpectation Maximization First introduced in 1977 Lots of
Lecture 15 Expectation Maximization EM Unsupervised Learning andLecture 15 Expectation Maximization EM Unsupervised Learning and
Learning Sequence Motif Models Using Expectation Maximization EMLearning Sequence Motif Models Using Expectation Maximization EM
Learning Sequence Motifs Using Expectation Maximization EM andLearning Sequence Motifs Using Expectation Maximization EM and