Multiagent Systems Solution Concepts for Normal Form Matrix

Multiagent Systems Solution Concepts for Normal Form (Matrix) Games © Manfred Huber 2018 1

Solution Concepts n n Solution concepts define “interesting” outcomes in a multiagent game, e. g. : n Pareto-optimal outcomes n Pure-strategy Nash equilibrium n Mixed-strategy Nash equilibrium n Maxmin strategy profile n Minmax strategy profile Important questions n Do they exist ? n How can they be computed ? © Manfred Huber 2018 2

Zero-Sum Games n Zero-Sum games are games where all the sum of the payoffs for all agents for all strategies is equal to 0. n n Any game in which a transformation of the utility function of the form u’ = k*u + l, k>0, l≥ 0 leads to a zero-sum game can itself be considered a zersum game. a 2 -payer zero-sum game is completely antagonistic n © Manfred Huber 2018 u 1 = -u 2 3

Nash Equilibria in 2 -Player Zero-Sum Games n In 2 -player zero-sum games Nash equilibria are minmax and maxmin strategies n Solution can be found by searching through the possible sets of support for mixed strategies n n Exponential complexity Maxmin solution can be formulated as a linear program n © Manfred Huber 2018 Maxmin for agent 1 is equal to Minmax for agent 2 4

Linear Programming n Linear programs are optimization problems under linear, closed inequality constraints n xi are the variables to solve for © Manfred Huber 2018 5

2 -Player Zero-Sum Game n In 2 -player zero-sum games, maxmin and minmax can be expressed as linear programs n maxmin for agent 1 or minmax for agent 2: © Manfred Huber 2018 6

2 -Player Zero-Sum Game n minmax for agent 1 or maxmin for agent 2: © Manfred Huber 2018 7

Maxmin Strategies for General Games n maxmin strategies can be computed for general sum games n n Computation by re-designing the game n n n maxmin strategy determines the agent’s strategy only based on its utility function Construct a zero-sum game G’ by maintaining the agent’s payoffs and changing the other agent’s payoff function The maxmin solution for the agent in the modified game G’ is the same as for the original game maxmin strategy in a general sum game is not necessarily a Nash equilibrium © Manfred Huber 2018 8

Nash Equilibria for 2 -Player Non Zero-Sum Games n For non zero-sum games, linear programming does no longer work because there is no single objective function n The complexity of computing Nash equilibria in general sum games is unknown. n n It is assumed that it is worst case exponential 2 -Player general sum games can be fomulated as Linear Complementarity Problems (LCP) n n © Manfred Huber 2018 LCP is a constraint satisfaction problem without an objective function Formulation considers both players’ utility functions 9

LCP Formulation for 2 -Player Non Zero-Sum Games n LCP combines the constraints of both agents and adds complementarity constraints © Manfred Huber 2018 10

Nash Equilibria for 2 -Player Non Zero-Sum Games n LCP for 2 -player games can be solved, e. g. , using the Lemke-Howson algorithm n n n Moves along the edges of a labeled graph in strategy space Worst-case exponential Other solution is to search the space of supports n n Determine which sets of actions could yield mixed Nash equilibria Compute the corresponding probabilities and verify that it is an equilibrium © Manfred Huber 2018 11

Domination and Strategy Removal n Strategy profiles can be related n si strictly dominates si’ for player i if n si weakly dominates si’ for player i if n si very weakly dominates si’ for player i if © Manfred Huber 2018 12

Domination and Nash Equilibria n n A dominant strategy is a strategy that dominates all others A strategy profile consisting of dominant strategies for all players must be a Nash equilibrium © Manfred Huber 2018 13

Iterated Removal of Dominated Strategies n No equilibrium can be strictly dominated by another strategy n n All strictly dominated strategies can be removed while still maintaining the solution to the game Iterated removal of dominated strategies repeats the removal process until no further dominated strategies are available © Manfred Huber 2018 14

Iterated Removal of Dominated Strategies n n L C R U 3, 1 0, 0 M 1, 1 5, 0 D 0, 1 4, 1 0, 0 R is dominated by L for player 2 M is dominated by [0. 5: U; 0. 5: D] for player 1 © Manfred Huber 2018 L C R U 3, 1 0, 0 M 1, 1 5, 0 D 0, 1 4, 1 0, 0 15

Iterated Removal of Dominated Strategies n Iterated removal preserves Nash equilibria n n Strict dominance preserves all equilibria Weak or very weak dominance preserves at least one equilibrium Removal order can influence which equilibria are preserved Iterated removal can be used as a preprocessing step for Nash equilibrium calculation © Manfred Huber 2018 16

Correlated Equilibria n In particular in coordination games the Nash equilibrium does not achieve a very good performance n n Equilibrium results in strategies that often end up in low payoff outcomes go wait go -100, -100 10, 0 wait 0, 10 -10, -10 What could solve this ? © Manfred Huber 2018 17

Correlated Equilibria n A solution would be to coordinate the random picks by the players n n Has to be outside the control and insight of each player or they could change their strategy Central random variable with a common, known distribution and a private signal to each of the players n © Manfred Huber 2018 Signal is correlated to other signals but does not determine the other players’ signals 18

Correlated Equilibria n A correlated equilibrium is a tuple (v, π, σ) n n v is a vector of random variables with domains D π is the joint distribution of v σ is a vector of mappings from D to actions in A for every mapping σ’ : © Manfred Huber 2018 19

Correlated Equilibria n For every Nash equilibrium there exists a corresponding correlated equilibrium. n n If the mapping is replaced by the decoupled probabilistic choices and the mapping is reduced to the action choice indicated by the domain, the correlated equilibrium reduces to a Nash equilibrium Not every correlated equilibrium is a Nash equilibrium © Manfred Huber 2018 20

Computing Correlated Equilibria n Linear programming constraints for CE n Objective function: e. g. social-welfare Not necessarily a Nash Equilibrium © Manfred Huber 2018 21

Computing Correlated Equilibria n CE are easier to compute than Nash Equilibria n n Only one randomization in CE and product of independent probabilities in NE Constraints for NE: This is a nonlinear constraint – No linear program © Manfred Huber 2018 22

Computing Nash Equilibria n No algorithm is known to compute Nash equilibria for n-player general sum games in polynomial time n n A number of iterated algorithms provide good approximations Multiagent learning can lead to efficient approximate solutions © Manfred Huber 2018 23