AlImam Mohammad Ibn Saud University CS 433 Modeling

Al-Imam Mohammad Ibn Saud University CS 433 Modeling and Simulation Lecture 06 – Part 03 Discrete Markov Chains 12 Apr 2009 Dr. Anis Koubâa

Classification of States: 1 2 A path is a sequence of states, where each transition has a positive probability of occurring. State j is reachable (or accessible) ( )ﻳﻤﻜﻦ ﺍﻟﻮﺻﻮﻝ ﺇﻟﻴﻪ from state i (i j) if there is a path from i to j –equivalently Pij (n) > 0 for some n≥ 0, i. e. the probability to go from i to j in n steps is greater than zero. States i and j communicate (i j) ( )ﻳﺘﺼﻞ if i is reachable from j and j is reachable from i. (Note: a state i always communicates with itself) A set of states C is a communicating class if every pair of states in C communicates with each other, and no state in C communicates with any state not in C.

Classification of States: 1 3 A state i is said to be an absorbing state if pii = 1. A subset S of the state space X is a closed set if no state outside of S is reachable from any state in S (like an absorbing state, but with multiple states), this means pij = 0 for every i S and j S A closed set S of states is irreducible ( )ﻏﻴﺮ ﻗﺎﺑﻞ ﻟﻠﺘﺨﻔﻴﺾ if any state j S is reachable from every state i S. A Markov chain is said to be irreducible if the state space X is irreducible.

Example 4 Irreducible Markov Chain p 01 p 00 0 p 12 1 p 10 Reducible Markov Chain p 01 p 00 0 p 10 p 22 2 p 21 p 12 1 p 23 2 p 32 3 p 14 Absorbing State 4 p 22 Closed irreducible set p 33

Classification of States: 2 5 State i is a transient state ( )ﺣﺎﻟﺔ ﻋﺎﺑﺮﺓ if there exists a state j such that j is reachable from i but i is not reachable from j. A state that is not transient is recurrent ( )ﺣﺎﻟﺔ ﻣﺘﻜﺮﺭﺓ. There are two types of recurrent states: 1. Positive recurrent: if the expected time to return to the state is finite. 2. Null recurrent (less common): if the expected time to return to the state is infinite (this requires an infinite number of states). A state i is periodic with period k >1, if k is the smallest number such that all paths leading from state i back to state i have a multiple of k transitions. A state is aperiodic if it has period k =1. A state is ergodic if it is positive recurrent and aperiodic.

Classification of States: 2 6 Example from Book Introduction to Probability: Lecture Notes D. Bertsekas and J. Tistsiklis – Fall 200

Transient and Recurrent States 7 q q We define the hitting time Tij as the random variable that represents the time to go from state j to stat i, and is expressed as: q k is the number of transition in a path from i to j. q Tij is the minimum number of transitions in a path from i to j. We define the recurrence time Tii as the first time that the Markov Chain returns to state i. q The probability that the first recurrence to state i occurs at the nth-step is q Ti Time for first visit to i given X 0 = i. q The probability of recurrence to state i is

Transient and Recurrent States 8 q The mean recurrence time is q A state is recurrent if fi=1 q q If Mi < then it is said Positive Recurrent If Mi = then it is said Null Recurrent q A state is transient if fi<1 q If i. , then is the probability of never returning to state

Transient and Recurrent States 9 q We define Ni as the number of visits to state i given X 0=i, Theorem: If Ni is the number of visits to state i given X 0=i, then q Proof Transition Probability from state i to state i after n steps

Transient and Recurrent States 10 The probability of reaching state j for first time in n-steps starting from X 0 = i. q The probability of ever reaching j starting from state i is

Three Theorems 11 If a Markov Chain has finite state space, then: at least one of the states is recurrent. If state i is recurrent and state j is reachable from state i then: state j is also recurrent. If S is a finite closed irreducible set of states, then: every state in S is recurrent.

Positive and Null Recurrent States 12 Let Mi be the mean recurrence time of state i A state is said to be positive recurrent if Mi<∞. If Mi=∞ then the state is said to be null-recurrent. Three Theorems If state i is positive recurrent and state j is reachable from state i then, state j is also positive recurrent. If S is a closed irreducible set of states, then every state in S is positive recurrent or, every state in S is null recurrent, or, every state in S is transient. If S is a finite closed irreducible set of states, then every state in S is positive recurrent.

Example 13 p 01 p 00 0 p 12 1 p 23 2 p 32 3 p 14 Transient States Recurrent State 4 p 22 Positive Recurrent States p 33

Periodic and Aperiodic States 14 Suppose that the structure of the Markov Chain is such that state i is visited after a number of steps that is an integer multiple of an integer d >1. Then the state is called periodic with period d. If no such integer exists (i. e. , d =1) then the state is called aperiodic. Example 1 0. 5 0 1 0. 5 Periodic State d = 2 2 1

Steady State Analysis 15 Recall that the state probability, which is the probability of finding the MC at state i after the kth step is given by: n An interesting question is what happens in the “long run”, i. e. , n This is referred to as steady state or equilibrium or stationary state probability n Questions: Do these limits exists? If they exist, do they converge to a legitimate probability distribution, i. e. , How do we evaluate πj, for all j.

Steady State Analysis 16 n Recall the recursive probability If steady state exists, then π(k+1) π(k), and therefore the steady state probabilities are given by the solution to the equations and n n If an Irreducible Markov Chain, then the presence of periodic states prevents the existence of a steady state probability Example: periodic. m

Steady State Analysis 17 n THEOREM: In an irreducible aperiodic Markov chain consisting of positive recurrent states a unique stationary state probability vector π exists such that πj > 0 and where Mj is the mean recurrence time of state j n The steady state vector π is determined by solving and n Ergodic Markov chain.

Discrete Birth-Death Example 18 1 -p p n 0 1 -p 1 1 -p i p p p Thus, to find the steady state vector π we need to solve and

Discrete Birth-Death Example 19 n In other words n Solving these equations we get n In general n Summing all terms we get

Discrete Birth-Death Example 20 n Therefore, for all states j we get n If p<1/2, then All states are transient n If p>1/2, then All states are positive recurrent

Discrete Birth-Death Example 21 n If p=1/2, then All states are null recurrent

Reducible Markov Chains 22 Transient Set T n n Irreducible Set S 1 Irreducible Set S 2 In steady state, we know that the Markov chain will eventually end in an irreducible set and the previous analysis still holds, or an absorbing state. The only question that arises, in case there are two or more irreducible sets, is the probability it will end in each set

Reducible Markov Chains 23 Transient Set T r s 1 sn Irreducible Set S i n Suppose we start from state i. Then, there are two ways to go to S. In one step or Go to r T after k steps, and then to S. n Define

Reducible Markov Chains 24 n n First consider the one-step transition Next consider the general case for k=2, 3, …