Best Reply Mechanisms Justin Thaler and Victor Shnayder

What are best-reply dynamics? • Start with an arbitrary strategy profile • In each step let some player switch his strategy to be a best reply to the current strategies of the others.

What are best-reply dynamics? Definition: A repeated-reply mechanism for a private info game G: • Extensive form game with perfect recall (same players) • At most M steps. In each step: • A single player announces an element of Ai • Players play in round-robin order • Stop when all players “pass” in n consecutive steps. • Enforce action proﬁle of the most recently announced actions • If M steps go by without stopping, penalize the players.

What are best-reply dynamics? • Need a penalty to ensure non-convergence is not in best interest of any player. • Realistic modeling assumption for BGP, TCP, etc. • Best-reply dynamics is the strategy profile of a repeated-reply mechanism in which each player i updates to i’s best-reply to the other players’ strategies each time it is i’s turn.

Why best reply dynamics? • If convergence occurs, we have a highly justifiable Nash Equilibrium • Computationally simple • Players only need private information • Feasible in distributed, asynchronous settings • Prescribed by existing protocols (Ex: BGP)

Why best reply dynamics? • In light of Theorems 1 and 2 (which we’ll see soon): • Often gives a non-VCG way of creating incentive compatible mechanisms (? ). And sometimes without $$$. • Often get collusion-proofness, Paretoefficiency

Outline • When do best reply dynamics work? • Universal max-solvability (UMS) • Thm: UMS implies convergence to unique NE, collusion-proofness • Example applications (correlated markets, BGP, etc) • Connections to strategy-proofness • Discussion

Universal maxdominance • A subset T of S is universally maxdominated if: • Very strong condition! • Existence of max-dominated set is strictly stronger than existence of dominated strategy. • Exists s , s ’ s. t. u (s , s ) < u (s ’, s ) for all s

Universal maxsolveability (UMS) • A game G is universally max-solvable if we can iteratively remove universally maxdominated strategy sets and get to a single strategy for each player. • Stronger condition than solvable by iterated removal of strictly dominated strategies (IRSDS)

Example 1 5, 5 0, 0 10, 0 4, 4 Solvable by IRSDS, but not UMS. Neither player has a universally max-dominated set. Note unique NE is not PE, and best-reply dynamics are not incentive compatible for the row player.

Example 2 0, 1 1, 0 UMS

Example 3 (UMS) L M R A 1, 9 2, 9 3, 1 B 3, 2 C 3, 1 4, 3 5, 4

Theorems Theorem 1: G is UMS ⇒ G has unique, pure NE, and it is collusion-proof. Corollary: Collusion-proof NE ⇒ NE is Pareto optimal Note that solvable by IRSDS suffices for unique, pure NE. UMS is needed for collusion-proofness and PE.

Proof of theorem 1: • By contradiction: G is UMS, so fix an elimination sequence of dominated strategy-sets. • Let s* be the final strategy profile. • If s* is not collusion proof NE, some set of players T can deviate and be better off. • Let s be new strategies where players in T change strategy from s* • Let si be first strategy eliminated. Then it was max-dominated, so si* is strictly better, so i can’t be better off.

Example 1 5, 5 0, 0 10, 0 4, 4 Solvable by IRSDS, but not UMS. Neither player has a universally max-dominated set. Note unique NE is not PE, and best-reply dynamics are not incentive compatible for the row player.

Theorems Theorem 2: If G is UMS with private information, then best reply dynamics are incentive-compatible in ex-post NE, and converge to the unique NE of the induced full-information game. Proof: Similar to Theorem 1. The main idea is that a strategy eliminated in the t‘th step of the UMS elimination process can never be used after the nt’th step of the best-reply mechanism.

Correlated two-sided markets • Agents: buyers and sellers • Game: weighted bipartite graph -- buyers on one side, sellers on the other • Buyers have preference order over sellers (higher edge weight = higher preference) • Sellers prefer buyers connected by heavier edges

Correlated two-sided markets are UMS • Let e be maximum weight edge. Choosing it universally max-dominates all other strategies of both endpoints. • Remove the two endpoints of e and all incident edges, repeat. • Therefore, best reply dynamics converge to ex-post NE.

Extended Example: BGP

Internet routing: BGP 12 d 1 d 2 d 1 2 d • Receive update messages from neighbours announcing routes to d. • Choose a single neighbor, whose route you prefer most, to send tra� c through. • Announce your new route to all your neighbors

Internet routing: BGP • BGP is asynchronous, distributed • Prescribes best-reply dynamics • But does BGP converge? • And is BGP “incentive compatible”? Do ASes have an incentive to deviate from the protocol?

Does BGP Converge? • We can break this into two questions: • Does a stable solution even exist in the static game? • If so, will BGP find such a solution? • But we only need one answer.

Does a Stable Solution Exist? 13 d 1 d No stable solution exists! 1 2 d 3 21 d 2 d It is actually NPcomplete to determine existence in 32 d general networks 3 d

Does BGP Converge When A Stable Solution Exists? • Notice that multiple NE exist. • And asynchronous best-reply dynamics do not necessarily converge. • So must not be UMS. 12 d 1 d 2 d 1 2 d

So What Do We Do? • Approach #1: Use mechanism design to achieve IC convergence, but solution must be distributed. • Approach #2: Identify conditions (on network topology and/or AS preferences) under which BGP converges and is IC. • Both approaches are canonical problems in Distributed Algorithmic Mechanism Design.

Approach #2 for Convergence • Griffin et al. (1999): If BGP fails to converge, then there exists a Dispute Wheel. • Each ui would rather route clockwise through ui+1 than Qi Image Source: Levin et al. “Internet Routing and Games, ” 2008.

Approach #2 for Convergence • Gao and Rexford (2001): Identified reasonable conditions based on economic structure of the Internet that guarantee No Dispute Wheel and hence convergence. (No bounds on convergence rate given). • But limited progress made until recently on conditions for guaranteeing that BGP is IC.

Approach #2 for Incentive Compatibility • Theorem 3: Assuming non-convergence after n 3 rounds is a penalty, and No Dispute Wheel holds, then routing games are UMS. • Corollary: Under the above conditions, bestreply strategies are IC in collusion-proof ex-post NE. • Corollary: Under the Gao-Rexford conditions, 3 BGP converges in O(n ) time and is IC.

Theorem 3 • Proof sketch: The case of finding the first universally max-dominated action set is general. • Find a node a 1 with at least 2 actions. Let R be a 1’s most preferred existing route. One of two cases must occur:

Theorem 3 1. Every node a 2 on R prefers the suffix of R leading from a 2 to d. In this case, if u is the closest node to d on R with at least two actions, then (u, d) universally maxdominates all other actions of u, and we’re done. 2. Some node a 2 on R prefers some other path over the suffix of R leading from a 2 to d. In this case, we repeat the analysis at a 2. Eventually we either form a dispute wheel or find ourselves in Case 1.

What’s left in Routing? • Complete characterization of BGP convergence (No Dispute Wheel sufficient, not necessary). • Conditions for convergence to globally optimal solution. Can it even be efficiently found? • Do mechanism design and/or $$$ have a role to play? • Changes in network topology?

Other applications • Congestion control • Criticism: Best-reply dynamics are only somewhat descriptive of how TCP works in practice. • Cost sharing games • Matching games (stable-roommate, intern assignment) • Auctions (unit demand bidders, • • GSP) Relies a lot on VCG results Main contribution is proof of convergence! (opposite of BGP)

Relationship to DSIC Given UMS game, best-replying is a strategy that gives ex-post NE. Play s(θ) θ Outcome Ex-post NE Get a direct-revelation, dominant strategy IC mechanism Good: New way to create DSIC mechanisms. Bad: Impossibility results limit the class of problems amenable to this approach (at least without money or limits on preferences).

Discussion • What is the main contribution? 1. Sufficient conditions for IC convergence of best-reply dynamics. General enough to encompass many applications, esp. BGP. 2. Bounds on time to convergence. 3. New framework for developing IC mechanisms?

Next Steps 1. Necessary conditions for best-reply dynamics to converge? To be IC (under what definition? )? 2. Better-reply dynamics? Other types of dynamics aka algorithms? What types of dynamics are reasonable or “natural”?

Economists and Complexity See recent blog post by Noam Nisan: Does complexity of equilibria matter? Kamal Jain: “If your laptop can’t find it then neither can the market“. Jeff Ely: “Solving the n-body problem is beyond the capabilities of the world’s smartest mathematicians. How do those rocks-for-brains planets manage to do pull it off? “