Introduction to Markov chains Examples of Markov chains

  • Slides: 23
Download presentation
Introduction to Markov chains Examples of Markov chains: - Random walk on a line

Introduction to Markov chains Examples of Markov chains: - Random walk on a line

Introduction to Markov chains Examples of Markov chains: - Random walk on a graph

Introduction to Markov chains Examples of Markov chains: - Random walk on a graph

Introduction to Markov chains Examples of Markov chains: - Random walk on a hypercube

Introduction to Markov chains Examples of Markov chains: - Random walk on a hypercube

Introduction to Markov chains Examples of Markov chains: - Markov chain on graph colorings

Introduction to Markov chains Examples of Markov chains: - Markov chain on graph colorings

Introduction to Markov chains Examples of Markov chains: - Markov chain on graph colorings

Introduction to Markov chains Examples of Markov chains: - Markov chain on graph colorings

Introduction to Markov chains Examples of Markov chains: - Markov chain on graph colorings

Introduction to Markov chains Examples of Markov chains: - Markov chain on graph colorings

Introduction to Markov chains Examples of Markov chains: - Markov chain on matchings of

Introduction to Markov chains Examples of Markov chains: - Markov chain on matchings of a graph

Definition of Markov chains - State space - Transition matrix P, of size |

Definition of Markov chains - State space - Transition matrix P, of size | |x| | - P(x, y) is the probability of getting from state x to state y - y 2 P(x, y) = This course: state space is finite

Definition of Markov chains - State space - Transition matrix P, of size |

Definition of Markov chains - State space - Transition matrix P, of size | |x| | - P(x, y) is the probability of getting from state x to state y - y 2 P(x, y) = This course: state space is finite

Definition of Markov chains Example: frog on lily pads

Definition of Markov chains Example: frog on lily pads

Definition of Markov chains Stationary distribution ¼ (on states in ): Example: frog with

Definition of Markov chains Stationary distribution ¼ (on states in ): Example: frog with unfair coins

Definition of Markov chains Other examples:

Definition of Markov chains Other examples:

Definition of Markov chains - aperiodic - irreducible - ergodic: a MC that is

Definition of Markov chains - aperiodic - irreducible - ergodic: a MC that is both aperiodic and irreducible Thm (3. 6 in Jerrum): An ergodic MC has a unique stationary distribution. Moreover, it converges to it as time goes to 1.

Gambler’s Ruin A gambler has k dollars initially and bets on (fair) coin flips.

Gambler’s Ruin A gambler has k dollars initially and bets on (fair) coin flips. If she guesses the coin flip: gets 1 dollar. If not, loses 1 dollar. Stops either when 0 or n dollars. Is it a Markov chain ? How many bets until she stops playing ? What are the probabilities of 0 vs n dollars ?

Gambler’s Ruin Stationary distribution ?

Gambler’s Ruin Stationary distribution ?

Gambler’s Ruin Starting with k dollars, what is the probability of reaching n ?

Gambler’s Ruin Starting with k dollars, what is the probability of reaching n ? What is the probability of reaching 0 ?

Gambler’s Ruin How many bets until 0 or n ?

Gambler’s Ruin How many bets until 0 or n ?

Coupon Collecting - n coupons - each time get a coupon at random, chosen

Coupon Collecting - n coupons - each time get a coupon at random, chosen from 1, …, n (i. e. , likely will get some coupons multiple times) - how long to get all n coupons ? Is this a Markov chain ?

Coupon Collecting How long to get all n coupons ?

Coupon Collecting How long to get all n coupons ?

Coupon Collecting How long to get all n coupons ?

Coupon Collecting How long to get all n coupons ?

Coupon Collecting Proposition: Let ¿ be the random variable for the time needed to

Coupon Collecting Proposition: Let ¿ be the random variable for the time needed to collect all n coupons. Then, Pr(¿ > d n log n + cn e) · e-c.

Random Walk on a Hypercube How many expected steps until we get to a

Random Walk on a Hypercube How many expected steps until we get to a random state ? (I. e. , what is the mixing time ? )

Random Walk on a Hypercube How many expected steps until we get to a

Random Walk on a Hypercube How many expected steps until we get to a random state ? (I. e. , what is the mixing time ? )

Markov Chains 1 Markov Chains A Markov ChainMarkov Chains 1 Markov Chains A Markov Chain
Tutorial 8 Markov Chains Markov Chains Consider aTutorial 8 Markov Chains Markov Chains Consider a
Markov Chains Summary n n Markov Chains DiscreteMarkov Chains Summary n n Markov Chains Discrete
Markov Chains Summary n n Markov Chains DiscreteMarkov Chains Summary n n Markov Chains Discrete
Markov Chains Markov chains describe the changes inMarkov Chains Markov chains describe the changes in
Markov Model Markov Model Markov Chain Firstorder MarkovMarkov Model Markov Model Markov Chain Firstorder Markov
Introduction to HMMs Markov chains and Hidden MarkovIntroduction to HMMs Markov chains and Hidden Markov
CREST Examples Basic Examples Function Examples Limitation ExamplesCREST Examples Basic Examples Function Examples Limitation Examples
CREST Examples Basic Examples Function Examples Limitation ExamplesCREST Examples Basic Examples Function Examples Limitation Examples
Crown Examples Basic Examples Function Examples Limitation ExamplesCrown Examples Basic Examples Function Examples Limitation Examples
CREST Examples Basic Examples Function Examples Limitation ExamplesCREST Examples Basic Examples Function Examples Limitation Examples
CREST Examples Basic Examples Function Examples Limitation ExamplesCREST Examples Basic Examples Function Examples Limitation Examples
Crown Examples Basic Examples Function Examples Limitation ExamplesCrown Examples Basic Examples Function Examples Limitation Examples
MARKOV CHAIN LONGRUN PROPERTIES OF MARKOV CHAINS LONGRUNMARKOV CHAIN LONGRUN PROPERTIES OF MARKOV CHAINS LONGRUN
Markov Chains and Hidden Markov Models stochastic generativeMarkov Chains and Hidden Markov Models stochastic generative
Markov Chains Markov Decision Processes MDP Reinforcement LearningMarkov Chains Markov Decision Processes MDP Reinforcement Learning
Markov Chains Plan Introduce basics of Markov modelsMarkov Chains Plan Introduce basics of Markov models
Markov Chains and Hidden Markov Models stochastic generativeMarkov Chains and Hidden Markov Models stochastic generative
Markov Chains What is a Markov Chain WeMarkov Chains What is a Markov Chain We
Markov Chains and Hidden Markov Models stochastic generativeMarkov Chains and Hidden Markov Models stochastic generative
Markov Chains H Plan Introduce basics of MarkovMarkov Chains H Plan Introduce basics of Markov
Markov Chains The overall picture Markov Process DiscreteMarkov Chains The overall picture Markov Process Discrete