Stochastic Games Mr Sujit P Gujar eEnterprise Lab

Stochastic Games Mr Sujit P Gujar. e-Enterprise Lab Computer Science and Automation IISc, Bangalore.

Agenda Stochastic Game n Special Class of Stochastic Games n Analysis : Shapley’s Result. n Applications n e-Enterprise Lab

Repeated Game n When players interact by playing a similar stage game (such as the prisoner's dilemma) numerous times, the game is called a repeated game. e-Enterprise Lab

Stochastic Game n n Stochastic game is repeated game with probabilistic/stochastic transitions. There are different states of a game. Transition probabilities depend upon actions of players. Two player stochastic game : 2 and 1/2 player game. e-Enterprise Lab

Repeated Prisoner’s Dilemma n Consider Game tree for PD repeated twice. Assume each player has the same two options at each info set: {C, D} subga me 1 2 1 1 2 2 What is Player 1’s strategy set? (Cross product of all choice sets at all information sets…) {C, D} x {C, D} 25 = 32 possible strategies e-Enterprise Lab First Iteratio n Second Iteratio n

Issues in Analyzing Repeated Games n How to we solve infinitely repeated games? n Strategies are infinite in number. n Need to compare sums of infinite streams of payoffs e-Enterprise Lab

Stochastic Game : The Big Match n n Every day player 2 chooses a number, 0 or 1 Player 1 tries to predict it. Wins a point if he is correct. This continues as long as player 1 predicts 0. But if he ever predicts 1, all future choices for both players are required to be the same as that day's choices. e-Enterprise Lab

The Big Match n n S = {0, 1*, 2*} : State space. s 1 ={0} s 2 ={1} n s 0 ={0, 1} N = {1, 2} n P 02 = 00 P = 0 0 0 1 1 P 01 = 1 0 0 0 n A = Payoff Matrix = 1* 0* 0 1 e-Enterprise Lab

n n The "Big-Match" game is introduced by Gillette (1957) as a difficult example. The Big Match David Blackwell; T. S. Ferguson The Annals of Mathematical Statistics, Vol. 39, No. 1. (Feb. , 1968), pp. 159 -163. e-Enterprise Lab

Scenario N Total number of States/Positions mk Choices for row player at position k nk Choices for column player at position k skij > 0 The probability with which the game in position k stops when player 1 plays i and player 2, j. pklij The probability with which the game in position k moves to l when player 1 plays i and player 2, j. s Min skij akij Payoff to row player in stage k. M Max |akij| e-Enterprise Lab

Stationary Strategies n n n Enumerating all pure and mixed strategies is cumbersome and redundant. Behavior strategies those which specify a player the same probabilities for his choices every time the same position is reached by whatever route. x = (x 1, x 2, …, x. N) each xk = (xk 1, xk 2, …, xkmk) e-Enterprise Lab

Notation n Given a matrix game B, n n val[B] = minimax value to the first player. X[B] = The set of optimal strategies for first player. Y[B] = The set of optimal strategies for second player. It can be shown, (B and C having same dimensions) |val[B] - val[C]| ≤ max |bij - cij| e-Enterprise Lab

n n When we start in position k, we obtain a particular game, We will refer stochastic game as, Define, e-Enterprise Lab

1 Shapley’s Results 1 L. S. Shapley, Stochastic Games. PNAS 39(1953) 1095 -1100 e-Enterprise Lab

n Let, denote the collection of games whose pure strategies are the stationary strategies of. The payoff function of these new games must satisfy, e-Enterprise Lab

Shapley’s Result, e-Enterprise Lab

Applications n n n 1 When N = 1, By setting all skij = s > 0, we get model of infinitely repeated game with future payments are discounted by a factor = (1 -s). If we set nk = 1 for all k, the result is “dynamic programming model”. 1 von Neumann J. , Ergennise eines Math, Kolloquims, 8 73 -83 (1937) e-Enterprise Lab

Example 1 -1 -2 1 n n n P 1 = 1 -s n 1 -s Consider 1 -s the game 1 -s with N = 1, n x=(0. 6, 0. 4) A= n y=(0. 4, 0. 6) P 2 = 1 -2 s 1 -s n n 1 -2 s x=(0. 61, 0. 39) y=(0. 39, 0. 61) e-Enterprise Lab

n Thank You!! e-Enterprise Lab