Extensiveform games and how to solve them CMU

Extensive-form games and how to solve them CMU 15 -381 and 15 -681 Fall 2017 Teachers: Noam Brown Tuomas Sandholm

What are Games? • Requirements for a game: • At least 2 players • Actions available for each player • Payoffs to each player for those actions • Examples?

What is Game Theory? • Assumptions in Game Theory: • 1) All players want to maximize their utility • 2) All players are rational • 3) It is common knowledge that all players are rational

Two-Player Zero-Sum Games • For now, we will focus on two-player zero-sum games • Only two players • Whatever one player wins, the other player loses • Common in recreational games (rock-paper-scissors, chess, Go, poker) Player 1 Player 2 Rock Paper Scissors Rock 0, 0 -1, 1 1, -1 Paper 1, -1 0, 0 -1, 1 Scissors -1, 1 1, -1 0, 0

Extensive-Form Games P 1 • Moves are done sequentially • Although “information sets”, discussed later, enable modeling simultaneous moves too • There can be an extra player called chance (aka. nature) that plays stochastically, not strategically • Game forms a tree • Rewards are shown for P 1, then P 2 -4, 4 P 2 -2, 2 -3, 3 P 1 1, -1 0, 0

Backward Induction P 1 • Start at the bottom of the tree and solve assuming we reached that point • Replace the decision point with the value of the decision • Move up, and repeat P 2 -4, 4 P 2 -2, 2 -3, 3 P 1 1, -1 0, 0

Backward Induction P 1 • Start at the bottom of the tree and solve assuming we reached that point • Replace the decision point with the value of the decision • Move up, and repeat P 2 -4, 4 P 2 -2, 2 -3, 3 P 1 1, -1 0, 0

Backward Induction P 1 • Start at the bottom of the tree and solve assuming we reached that point • Replace the decision point with the value of the decision • Move up, and repeat P 2 2, 4 P 2 -2, 2 5, 3 1, -1

Backward Induction P 1 • Start at the bottom of the tree and solve assuming we reached that point • Replace the decision point with the value of the decision • Move up, and repeat P 2 2, 4 P 2 -2, 2 5, 3 1, -1

Backward Induction P 1 • Start at the bottom of the tree and solve assuming we reached that point • Replace the decision point with the value of the decision • Move up, and repeat P 2 -4, 4 -2, 2 -3, 3

Backward Induction • Start at the bottom of the tree and solve assuming we reached that point • Replace the decision point with the value of the decision • Move up, and repeat P 1 -4, 4 -2, 2

Backward Induction P 1 • Start at the bottom of the tree and solve assuming we reached that point • Replace the decision point with the value of the decision • Move up, and repeat P 2 -4, 4 P 2 -2, 2 -3, 3 P 1 1, -1 0, 0

Converting to Normal Form P 1 A • Every extensive-form game can be converted to a normal-form game L • Must define strategy for every situation P 2 -2, 2 B R -3, 3 0, 0 P 2 -1, 1 • Problem: Exponential in the size of the game A -2, 2 -3, 3 B 0, 0 -1, 1

Solving Perfect-Information Games • Checkers solved – can always draw [Schaeffer et al. "Checkers is solved. " Science (2007)] • At about 1020 states, checkers is the largest game ever solved. Chess has about 1040. Go has about 10170. 14

Imperfect-information Games s C d a He y Pla y P 1 -0. 5 ds ds He a 1 ils 1 P 2 Ta • An information set is the set of all possible states (nodes in the game tree) that the player may be in, given the information available to him. -1 ll Se P 1 ll Se • Backward induction doesn’t work in imperfectinformation games! s Pla • What about games like poker? Ta il -1 • A strategy must be identical for all nodes in an information set. 15

Half-Street Kuhn Poker • Poll: What should P 1 do with a Queen? • • P 2 Fold Call • - If P 2 folds, P 1 wins 1 chip • - If P 2 calls, the higher card wins 2 chips Check • If P 1 checks, the higher card wins 1 chip • If P 1 bets, P 2 may call or fold P 1 Bet • 3 cards: K, Q, and J. Highest card wins • Two players are dealt one of the cards • Both players ante 1 chip • P 1 may bet or check 1. Always bet 2. Sometimes bet, sometimes check 3. Always check 4. No idea 16

Imperfect-information Games C Check Fold 1 -2 Fold Check Fold Check -2 Call Fold Check Bet 1 P 2 Call 2 P 2 Call 1 -1 P 2 Call 2 P 1 -1 P 2 Call 1 : Q Bet -1 P 2 : K P 1: J, P 2 P 1 Bet Bet P 2 J, P 2 P 1 1 1 2 K 2: P 1 1 1 P 1: , P P 1 , : Q P 2: , K : P 1 J P 1 J Q 2: P 1: K, P : P 2 1 -2 Try iterative removal of dominated strategies! 17

Imperfect-information Games C K 2: Fold -2 -2 Fold Check Fold P 2 Call 1 -1 Call 2 -1 P 2 Bet P 2 Call 1 -1 P 1 Bet P 2 : Q P 1 Bet Bet 1 : K P 1: J, P 2 Check P 1 P 1 J, P 2 Check P 1 P P 1: , P , : Q P 1 J P 2: , K : 1 J P 1 Q 2: P 1: K, P : P 2 1 -2 18

Imperfect-information Games C Check P 1 -1 -1 P 2 -2 Fold P 1 Call 1 : Q Bet 2 -1 : K P 1: J, P 2 Call 1 P 2 J, P 2 Bet Bet 1 P 1: P 1 Check P 1 P 2 K 2: P 1 P , : Q P 1 J P 2: , K : 1 J P 1 Q 2: P 1: K, P : P 2 1 -2 19

Imperfect-information Games • Two questions remain: • How often should P 1 bet with a Jack (bluff)? • How often should P 2 call with a Queen when P 1 bets? Call with Q Bet with J Check with J Fold with Q

Imperfect-information Games • 21

Solving two-player zero-sum extensive-form games

Alpha. Go techniques extend, in principle, to other perfect-information games

Perfect-Information Games

Perfect-Information Games

Perfect-Information Games

Imperfect-Information Games

Imperfect-Information Games

Example Game: Coin Toss P 1 Information Set 0. 5 P = ads C P = Tail s He P 1 . 5 P 1 Se ll 0 = EV P 2 He ad s He 1 ils 1 Ta ils Ta ad s P 2 -1 P 2 Information Set y Pla Se ll 5 Play E V . 0 = 0. 5 -1

Imperfect-Information Games: Coin Toss s ead C P = Tail s H Se ll 5 . -0 P 2 He ad s He 1 ils 1 Ta ils Ta ad s P 2 -1 Imperfect-Information Subgame P 1 y Pla = EV Play = EV P 1 5 0. 5 Se ll . 5 0 P = -1

Imperfect-Information Games: Coin Toss C . 5 0 P = s ead P = Tail s H Se ll P 1 . 5 1. 0 = y Pla 0 = EV P 2 1 P He = 1 ad. 0 s 1 0 0. P = ails T P He= 1 ad. 0 s P 2 -1 EV P 1 5 0 -1. = EV Play EV . 0 = 0. 5 -1

Imperfect-Information Games: Coin Toss Heads EV = 0. 5 Tails EV = 1. 0 Average = 0. 75 C . 5 0 P = s ead P = Tail s H P 1 . 5 Se ll ll Se 0 = EV P 2 1 P He = 1 ad. 0 s 1 0 0. P = ails T P He= 1 ad. 0 s P 2 -1 . 0 =1 EV lay P EV 0 -1. = EV Play P 1 5 . 0 = 0. 5 -1

Imperfect-Information Games: Coin Toss C . 5 0 P = s ead P = Tail s H P 1 . 5 Se ll 0 = EV P 2 1 P He = 0 ad. 0 s 1 0 1. P = ails T P He= 0 ad. 0 s P 2 -1 0 -1. = EV Play P 1 5 1. 0 = EV Play EV . 0 = 0. 5 -1

Imperfect-Information Games: Coin Toss C . 5 0 P = s ead P = Tail s H EV P 1 ll Se ll 5 0. = P 2 1 P He = 0 ad. 0 s 1 0 1. P = ails T P He= 0 ad. 0 s P 2 -1 0 -1. = EV Play P 1 5 . 0 =1 EV Play EV . 0 = 0. 5 Se Heads EV = 1. 0 Tails EV = -0. 5 Average = 0. 25 -1

Imperfect-Information Games: Coin Toss C . 5 0 P = s ead P = Tail s H P 1 . 5 Se ll 0 = EV P 2 1 P He= 0. ad 25 s 1 75 0. s P = Tail P He= 0. ad 25 s P 2 -1 5 -0. = EV Play P 1 5 0. 5 = EV Play EV . 0 = 0. 5 -1

Imperfect-Information Games: Coin Toss Heads EV = 0. 5 Tails EV = -0. 5 Average = 0. 0 C . 5 0 P = s ead P = Tail s H P 1 . 5 Se ll 0 = EV P 2 1 P He= 0. ad 25 s 1 75 0. s P = Tail P He= 0. ad 25 s P 2 -1 5 -0. = EV Play P 1 5 0. 5 = EV Play EV . 0 = 0. 5 -1

Imperfect-Information Games: Coin Toss. 5 0 P = s ead C P = Tail s H P 1 . 5 Se ll 0 = EV y Pla P 2 1 1 75 0. s P = Tail -1 P He= 0. ad 25 s P 2 P He= 0. ad 25 s P 1 5 Play EV . 0 = 0. 5 -1

Imperfect-Information Games: Coin Toss. 5 0 P = s ead C P = Tail s H EV . 0 = y Pla Se ll P 1 5 Play P 2 1 1 75 0. s P = Tail -1 P He= 0. ad 25 s P 2 P He= 0. ad 25 s EV P 1 Se ll . 5 0 =- 0. 5 -1

Imperfect-Information Games: Coin Toss. 5 0 P = s ead C P = Tail s H EV . 0 = y Pla Se ll P 1 5 Play P 2 1 1 25 0. s P = Tail -1 P He= 0. ad 75 s P 2 P He= 0. ad 75 s EV P 1 Se ll . 5 0 =- 0. 5 -1

Increasing scalability: Abstraction • Idea: Create smaller game that is similar to big game • Run equilibrium-finding algorithm on smaller game • Map computed equilibrium to a strategy profile in big game

Abstraction [Gilpin & Sandholm EC-06, J. of the ACM 2007…] Original game Abstracted game Compact representation Automated abstraction 161 10 Equilibrium-finding algorithm Reverse mapping ε-equilibrium Foreshadowed by Shi & Littman 01, Billings et al. IJCAI-03

Lossless abstraction C C 0. 5 P 1 P 2 10 0 P 1 P 2 0 1 P 2 10 10 P 2 0 0 10

Lossy abstraction C C 0. 5 P 1 P 2 8 0 P 1 P 2 0 1 P 2 10 10 P 2 0 0 P 2 10 9 P 2 0 0 10

ε-Nash equilibrium • Standard Nash equilibrium: • • Each player has no incentive to deviate In other words: No agent can gain utility by switching to another strategy • ε-Nash equilibrium: • • Each player has only ε incentive to deviate In other words: No agent can gain more than ε utility by switching to another strategy

Increasing scalability: Sparse algorithms •

Plan for next part of the presentation • We will introduce counterfactual regret minimization (CFR)-based equilibrium finding in two-player zero-sum games • Most common algorithm • Relatively easy to understand implement • We start from first principles and proceed step by step to the most advanced algorithms in this family • Used to be best in practice, but now there is a gradient-based algorithm that is better in practice also [Kroer et al. EC-17] • But that is much more difficult to understand