MultiAgent Planning in Complex Uncertain Environments Daphne Koller

Multi-Agent Planning in Complex Uncertain Environments Daphne Koller Stanford University Joint work with: Carlos Guestrin (CMU) Ronald Parr (Duke)

Collaborative Multiagent Planning Long-term goals n n n Multiple agents Coordinated decisions Search and rescue, firefighting Factory management Multi-robot tasks (Robosoccer) Network routing Air traffic control Computer game playing © 2004 – Carlos Guestrin, Daphne Koller Collaborative Multiagent Planning

Joint Planning Space n n Joint action space: n Each agent i takes action ai at each step n Joint action a= {a 1, …, an} for all agents Joint state space: n Assignment x 1, …, xn to some set of variables X 1, …, Xn n Joint state x= {x 1, …, xn} of entire system Joint system: Payoffs and state dynamics depend on joint state and joint action Cooperative agents: Want to maximize total payoff © 2004 – Carlos Guestrin, Daphne Koller

Exploiting Structure n Real-world problems have: n n n Hundreds of objects Googles of states Real-world problems have structure! Approach: Exploit structured representation to obtain efficient approximate solution © 2004 – Carlos Guestrin, Daphne Koller

Outline n Action Coordination n n Joint Planning n n Factored Value Functions Coordination Graphs Context-Specific Coordination Multi-Agent Markov Decision Processes Efficient Linear Programming Solution Decentralized Market-Based Solution Generalizing to New Environments n n Relational MDPs Generalizing Value Functions © 2004 – Carlos Guestrin, Daphne Koller

One-Shot Optimization Task n n n Q-function Q(x, a) encodes agents’ payoff for joint action a in joint state x Agents’ task: To compute #actions is exponential Complete state observability Full agent communication © 2004 – Carlos Guestrin, Daphne Koller

Factored Payoff Function n n Approximate Q function as sum of Q sub-functions Each sub-function depends on local part of system n n n Two interacting agents Agent and important resource Two inter-dependent pieces of machinery Q(A 1, …, A 4, X 1, …, X 4) ¼ Q 1(A 1, A 4, X 1, X 4) + Q 2(A 1, A 2, X 1, X 2) + Q 3(A 2, A 3, X 2, X 3) + Q 4(A 3, A 4, X 3, X 4) © 2004 – Carlos Guestrin, Daphne Koller [K. & Parr ’ 99, ’ 00] [Guestrin, K. , Parr ’ 01]

Distributed Q Function n Q sub-functions assigned to relevant agents Q(A 1, …, A 4, X 1, …, X 4) ¼ Q 1(A 1, A 4, X 1, X 4) + Q 2(A 1, A 2, X 1, X 2) + Q 3(A 2, A 3, X 2, X 3) + Q 4(A 3, A 4, X 3, X 4) 2 3 1 4 [Guestrin, K. , Parr ’ 01] © 2004 – Carlos Guestrin, Daphne Koller Q 4

Multiagent Action Selection Instantiate current state x Distributed Q function Maximal action argmaxa Q 2(A 1, A 2, X 1, X 2) 2 Q 1(A 1, A 4, X 1, X 4) 3 1 4 Q 4(A 3, A 4, X 3, X 4) © 2004 – Carlos Guestrin, Daphne Koller Q 3(A 2, A 3, X 2, X 3)

Instantiating State x Limited observability: agent i only observes variables in Qi Q 2(A 1, A 2, X 1, X 2) 2 Q 1(A 1, A 4, X 1, X 4) 3 1 4 Q 4(A 3, A 4, X 3, X 4) © 2004 – Carlos Guestrin, Daphne Koller Observe only X 1 and X 2 Q 3(A 2, A 3, X 2, X 3)

Choosing Action at State x Instantiate current state x Maximal action maxa Q 2(A 1, A 2), X 1, X 2) 2 Q 1(A 1, A 4), X 1, X 4) 3 1 4 Q 4(A 3, A 4), X 3, X 4) © 2004 – Carlos Guestrin, Daphne Koller Q 3(A 2, A 3), X 2, X 3)

Variable Elimination + + + maxa Use variable elimination for maximization: max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + Q 3 ( A 2 , A 3 ) + Q 4 ( A 3 , A 4 ) A 1 , A 2 , A 3 , A 4 , A 2[Q) ( A , A ) + Q ( A , A ) ] = max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , Q + 1 max A 22)(A 3 2 3 4 A 1 , A 2 , A 4 A 3 = max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + g 1 ( A 2 , A 4 ) A 1 , A 2 , A 4 Q 1(A 1, A 4) n n A 2 A 4 Q A 3) 3(Aof 2, optimal Value A 3 action Limited communication Attack for optimal action choice Attack 5 Attack Defend 6 graph Comm. bandwidth = tree-width of coord. © 2004 – Carlos Guestrin, Daphne Koller Q 4(A 3, ADefend 4) Defend Attack 8 Defend 12

Choosing Action at State x max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + Q 3 ( A 2 , A 3 ) + Q 4 ( A 3 , A 4 ) A 1 , A 2 , A 3 , A 4 = max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + max[Q 3 ( A 2 , A 3 ) + Q 4 ( A 3 , A 4 ) ] A 1 , A 2 , A 4 A 3 = max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + g 1 ( A 2 , A 4 ) A 1 , A 2 , A 4 © 2004 – Carlos Guestrin, Daphne Koller

Choosing Action at State x max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + Q 3 ( A 2 , A 3 ) + Q 4 ( A 3 , A 4 ) A 1 , A 2 , A 3 , A 4 = max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + max[Q 3 ( A 2 , A 3 ) + Q 4 ( A 3 , A 4 ) ] A 1 , A 2 , A 4 A 3 = max Q 1 ( A 1 , A 4 ) + Q 2 ( A 1 , A 2 ) + g 1 ( A 2 , A 4 ) A 1 , A 2 , A 4 Q 2(A 1, A 2) 2 Q 1(A 1, A 4) 3 1 4 Q 4(A 3, A 4) © 2004 – Carlos Guestrin, Daphne Koller max A g[1 Q (A 3(A ) 3) + 2, 2 A, 4 A 3 ]

Coordination Graphs n n Communication follows triangulated graph Computation grows exponentially in tree width n n n Graph-theoretic measure of “connectedness” Arises in BNs, CSPs, … Cost exponential in worst case, fairly low for many real graphs A 5 A 1 A 7 A 8 A 6 A 9 A 2 A 4 A 3 © 2004 – Carlos Guestrin, Daphne Koller A 11 A 10

Context-Specific Interactions n Payoff structure can vary by context n n Agents A 1, A 2 both trying to pass through same narrow corridor Can use context-specific “value rules” <At(X, A 1), At(X, A 2), A 1 = fwd A 2 = fwd : -100> n Hope: Context-specific payoffs will induce context-specific coordination A 1 © 2004 – Carlos Guestrin, Daphne Koller X A 2

Context-Specific Coordination A 6 A 5 A 1 A 2 A 4 A 3 Instantiate current state: x = true © 2004 – Carlos Guestrin, Daphne Koller

Context-Specific Coordination A 6 A 5 Coordination structure varies based. A 1 on context A 2 A 4 A 3 © 2004 – Carlos Guestrin, Daphne Koller

Context-Specific Coordination A 6 A 5 Coordination structure varies 1 based on. Acommunication A 2 Maximizing out A 1 © 2004 – Carlos Guestrin, Daphne Koller A 4 A 3 Rule-based variable elimination [Zhang & Poole ’ 99]

Context-Specific Coordination A 6 A 5 Coordination structure varies 1 based on. Aagent decisions A 2 A 4 A 3 Eliminate A 1 from the Rule-based variable elimination [Zhang & Poole ’ 99] graph © 2004 – Carlos Guestrin, Daphne Koller

Robot Soccer Kok, Vlassis & Groen University of Amsterdam Uv. A Trilearn 2002 won German Open 2002, but placed fourth in Robocup-2002. “ … the improvements introduced in Uv. A Trilearn 2003 … include an extension of the intercept skill, improved passing behavior and especially the usage of coordination graphs to specify the coordination requirements between the different agents. ” © 2004 – Carlos Guestrin, Daphne Koller

Robo. Soccer Value Rules n n Coordination graph rules include conditions on player role and aspects of global system state Example rules for player i, in role of passer: Depends on distance of j to goal after move © 2004 – Carlos Guestrin, Daphne Koller

Uv. A Trilearn 2003 Results Round 1 Opponent Round 1 Mainz Rolling Brains (Germany) 4 -0 Iranians (Iran) 31 -0 Sahand (Iran) 39 -0 a 4 ty (Latvia) 25 -0 Helios (Iran) 2 -1 AT-Humboldt (Germany) 5 -0 ZJUBase (China) 6 -0 Aria (Iran) 6 -0 Hana (Japan) 26 -0 Zenit-New. ERA (Russia) 4 -0 Robo. Sina (Iran) 6 -0 Wright Eagle (China) 3 -1 Everest (China) 7 -1 Aria (Iran) 5 -0 Semi-final Brainstormers (Germany) 4 -1 Final Tsinghu. Aeolus (China) 4 -3 Round 2 Round 3 Uv. A Trilearn won • • German Open 2003 US Open 2003 Robo. Cup 2003 German Open 2004 © 2004 – Carlos Guestrin, Daphne Koller Score 177 -7

Outline n Action Coordination n n Joint Planning n n Factored Value Functions Coordination Graphs Context-Specific Coordination Multi-Agent Markov Decision Processes Efficient Linear Programming Solution Decentralized Market-Based Solution Generalizing to New Environments n n Relational MDPs Generalizing Value Functions © 2004 – Carlos Guestrin, Daphne Koller

peasant footman building © 2004 – Carlos Guestrin, Daphne Koller Real-time Strategy Game Peasants collect resources and build Footmen attack enemies Buildings train peasants and footmen

Planning Over Time Markov Decision Process (MDP) representation: n Action space: Joint agent actions a= {a 1, …, an} n State space: Joint state descriptions x= {x 1, …, xn} n Momentary reward function R(x, a) n Probabilistic system dynamics P(x’|x, a) © 2004 – Carlos Guestrin, Daphne Koller

Policy: (x) = a x 0 At state x, action a for all agents (x 0) = both peasants get wood x 1 (x 1) = one peasant gets gold, other builds barrack x 2 © 2004 – Carlos Guestrin, Daphne Koller (x 2) = Peasants get gold, footmen attack

Value of Policy Expected longterm reward starting from x Value: V (x) Start from x 0 (x 0) x 0 R(x 0) V (x 0) = E[R(x 0) + R(x 1) + 2 R(x 2) + 3 R(x 3) + 4 R(x 4) + ] x 1 (x 1) R(x 1) x 1’ R(x 1’) x 1’’ Future rewards discounted by [0, 1) (x 1’) (x 1’’) R(x. Koller © 2004 – Carlos Guestrin, Daphne 1’’) x 2 R(x 2) x 3 R(x 3) x 4 R(x 4)

Optimal Long-term Plan Optimal policy *(x) Optimal Q-function Bellman Equations: Optimal policy: © 2004 – Carlos Guestrin, Daphne Koller Q*(x, a)

Solving an MDP Solve Bellman equation Optimal value V*(x) Optimal policy *(x) Many algorithms solve the Bellman equations: n n Policy iteration [Howard ’ 60, Bellman ‘ 57] Value iteration [Bellman ‘ 57] Linear programming [Manne ’ 60] … © 2004 – Carlos Guestrin, Daphne Koller

LP Solution to MDP n n n One variable V (x) for each state One constraint for each state x and action a Polynomial time solution © 2004 – Carlos Guestrin, Daphne Koller

Are We Done? n Planning is polynomial in #states and #actions n #states exponential in number of variables n #actions exponential in number of agents Efficient approximation by exploiting structure! © 2004 – Carlos Guestrin, Daphne Koller

Structured Representation Time t APeasant t+1 Peasant P’ Gold G’ ABuild Footman AFootman Factored MDP P(F’|F, G, AB, AF) F’ [Boutilier et al. ’ 95] n n Enemy E’ R © 2004 – Carlos Guestrin, Daphne Koller State Dynamics Decisions Rewards Complexity of representation: Exponential in #parents (worst case)

Structured Value function ? Factored MDP Structure in Time t t+1 X X’ t+2 t+3 X’’’ Y Y’ Y’’’ Z Z’ Z’’’ R Factored MDP R Almost! R R Structure in V* Factored V often provides good approximate value function © 2004 – Carlos Guestrin, Daphne Koller * V

Structured Value Functions [Bellman et al. ‘ 63], [Tsitsiklis & Van Roy ‘ 96] [K. & Parr ’ 99, ’ 00] n Approximate V* as a factored value function n In rule-based case: n n n hi is a rule concerning small part of the system wi is the value associated with the rule Goal: find w giving good approximation V to V* Factored value function V = wi h i Factored Q function Q = Qi Can use coordination graph © 2004 – Carlos Guestrin, Daphne Koller

Approximate LP Solution V (xwi)h (x) x h ((x x)) ³ ì wi V Q (Qa, (xa), x) subject to : minimize : i i ii íi " xx, , aa î " n One variable wi for each basis function n n i i Polynomial number of LP variables One constraint for every state and action n Exponentially many LP constraints © 2004 – Carlos Guestrin, Daphne Koller

So What Now? [Guestrin, K. , Parr ’ 01] Exponentially many linear = one nonlinear constraint © 2004 – Carlos Guestrin, Daphne Koller

Variable Elimination Revisited [Guestrin, K. , Parr ’ 01] n Use Variable Elimination to represent constraints: Exponentially fewer constraints Polynomial LP for finding good factored approximation to V* © 2004 – Carlos Guestrin, Daphne Koller

Network Management Problem n n n Computer runs processes Computer status = {good, dead, faulty} Dead neighbors increase dying probability Ring of Rings Reward for successful processes Each Sys. Admin takes local action = {reboot, not reboot } Star © 2004 – Carlos Guestrin, Daphne Koller k-grid

Scaling of Factored LP Explicit LP Factored LP 2 n (n+1 -k)2 k k = tree-width number of constraints 40000 Explicit LP 30000 Factored LP k = 12 20000 k = 10 k=8 10000 k=5 0 2 4 © 2004 – Carlos Guestrin, Daphne Koller 6 8 10 12 number of variables 14 16 k=3

Multiagent Running Time Ring of rings Star pair basis Star single basis © 2004 – Carlos Guestrin, Daphne Koller

Strategic 2 x 2 Factored MDP model offline 2 Peasants, 2 Footmen, Enemy, Gold, Wood, Barracks ~1 million state/action pairs Factored LP computes value function online Q x © 2004 – Carlos Guestrin, Daphne Koller Coordination graph computes argmaxa Q(x, a) World a

Demo: Strategic 2 x 2 Guestrin, Koller, Gearhart & Kanodia © 2004 – Carlos Guestrin, Daphne Koller

Limited Interaction MDPs [Guestrin & Gordon, ’ 02] n Some MDPs have additional structure: n n Agents are largely autonomous Interact in limited ways n n M 1 R 1 e. g. , competing for resources R 1 X’ 1 A 1 X 2 h 1 h 2 X 3 R 3 M 2 © 2004 – Carlos Guestrin, Daphne Koller X’ 2 X 1 X 2 X 1 A 1 X 2 X’ 2 X 3 X’ 3 R 2 X’ 3 A 2 A 1 X’ 2 R 2 X’ 1 X 2 Can decompose MDP as set of agentbased MDPs, with limited interface X 1 R 3 A 2

Limited Interaction MDPs [Guestrin & Gordon, ’ 02] n n n In such MDPs, our LP matrix is highly structured Can use Dantzig-Wolfe LP decomposition to solve LP optimally, in a decentralized way Gives rise to a market-like algorithm with multiple agents and a centralized “auctioneer” © 2004 – Carlos Guestrin, Daphne Koller

Auction-style planning n n Each agent solves local (stand-alone) MDP Agents send `constraint messages’ to auctioneer: n n n [Guestrin & Gordon, ’ 02] Auctioneer Must agree on “policy” for shared variables Auctioneer sends `pricing messages’ to agents n Set pricing based on conflicts Pricing reflects penalties for constraint violations Influences agents’ rewards in their MDP © 2004 – Carlos Guestrin, Daphne Koller Plan, plan $ $ Plan, plan, plan

Fuel Allocation Problem UAV start n n n UAVs share a pot of fuel Targets have varying priority Ignore target interference Bererton, Gordon, Thrun & Khosla © 2004 – Carlos Guestrin, Daphne Koller Target

Fuel Allocation Problem [Bererton, Gordon, Thrun, & Khosla , ’ 03] © 2004 – Carlos Guestrin, Daphne Koller

High-Speed Robot Paintball Bererton, Gordon & Thrun © 2004 – Carlos Guestrin, Daphne Koller

High-Speed Robot Paintball Game variant 1 Game variant 2 Coordination point x = start location Sensor Placement + = goal location © 2004 – Carlos Guestrin, Daphne Koller

High-Speed Robot Paintball Bererton, Gordon & Thrun © 2004 – Carlos Guestrin, Daphne Koller

Outline n Action Coordination n n Joint Planning n n Factored Value Functions Coordination Graphs Context-Specific Coordination Multi-Agent Markov Decision Processes Efficient Linear Programming Solution Decentralized Market-Based Solution Generalizing to New Environments n n Relational MDPs Generalizing Value Functions © 2004 – Carlos Guestrin, Daphne Koller

Generalizing to New Problems Many problems are “similar” Solve Problem 1 Solve Problem 2 Solve Problem n Good solution to Problem n+1 MDPs are different! Different sets of states, action, reward, transition, … © 2004 – Carlos Guestrin, Daphne Koller

Generalizing with Relational MDPs “Similar” domains have similar “types” of objects Relational MDP Exploit similarities by computing generalizable value functions Generalization Avoid need to replan Tackle larger problems © 2004 – Carlos Guestrin, Daphne Koller

Relational Models and MDPs [Guestrin, K. , Gearhart & Kanodia ‘ 03] n Classes: n n Relations n n Collects, Builds, Trains, Attacks… Instances n n Peasant, Footman, Gold, Barracks, Enemy… Peasant 1, Peasant 2, Footman 1, Enemy 1… Builds on Probabilistic Relational Models © 2004 – Carlos Guestrin, Daphne Koller [K. & Pfeffer ‘ 98]

Relational MDPs [Guestrin, K. , Gearhart & Kanodia ‘ 03] Enemy Footman Health H’ my_enemy Health AFootman R Count n Class-level transition probabilities depends on: n n Attributes; Actions; Attributes of related objects Class-level reward function Very compact representation! Does not depend on # of objects © 2004 – Carlos Guestrin, Daphne Koller H’

World is a Large Factored MDP Relational MDP n Instantiation (world): n n n # of objects Links between objects # instances of each class Links between instances Well-defined factored MDP © 2004 – Carlos Guestrin, Daphne Koller Factored MDP

MDP with 2 Footmen and 2 Enemies Footman 1 F 1. Health F 1. H’ F 1. A Enemy 1 E 1. Health E 1. H’ R 1 Footman 2 F 2. Health F 2. H’ F 2. A Enemy 2 R 2 © 2004 – Carlos Guestrin, Daphne Koller E 2. Health E 2. H’

World is a Large Factored MDP Relational MDP n n n # of objects Links between objects Instantiate world Well-defined factored MDP Use factored LP for planning We have gained nothing! © 2004 – Carlos Guestrin, Daphne Koller Factored MDP

Class-level Value Functions Footman 1 Enemy 1 F 1. Health E 1. Health Footman 2 F 2. Health V VF 2 VE 1. H)F F (E 1. H) E (F 2. H, E 2 V are Interchangeable! + + V (F 1. H, E 1. H, F 2. H, EUnits. H ) = 2 VF 1(F 1. H, E 1. H) VF 1 VF 2 VF VE 1 VE 2 VE Enemy 2 E 2. Health VE 2(EV 2. H E) + At state x, each footman has different contribution to V Given w. C — can instantiate value function for any world © 2004 – Carlos Guestrin, Daphne Koller

Factored LP-based Generalization How many samples? E 1 F 1 E 2 F 2 Sample Set I © 2004 – Carlos Guestrin, Daphne Koller Classlevel factored LP VF VE E 1 F 1 E 2 F 2 E 3 F 3 Generalize

Sampling Complexity Exponentially many worlds # objects in a world is unbounded need exponentially many samples? must sample very large worlds? NO! © 2004 – Carlos Guestrin, Daphne Koller

Theorem Sample m small worlds of up to O( ln 1/ ) objects m= samples Value function within O( ) of class-level value function optimized for all worlds, with prob. at least 1 - © 2004 – Carlos Guestrin, Daphne Koller Rcmax is the maximum class reward

Strategic 2 x 2 offline Relational MDP model 2 Peasants, 2 Footmen, Enemy, Gold, Wood, Barracks ~1 million state/action pairs Factored LP computes value function online Q x © 2004 – Carlos Guestrin, Daphne Koller Coordination graph computes argmaxa Q(x, a) World a

Strategic 9 x 3 offline Relational MDP model 9 Peasants, 3 Footmen, Enemy, Gold, Wood, Barracks ~3 trillion state/action pairs grows exponentially in # agents Factored LP computes value function online Qo x © 2004 – Carlos Guestrin, Daphne Koller Coordination graph computes argmaxa Q(x, a) World a

Strategic Generalization offline Relational MDP model 2 Peasants, 2 Footmen, Enemy, Gold, Wood, Barracks ~1 million state/action pairs 9 Peasants, 3 Footmen, Factored LP computes Enemy, Gold, Wood, Barracksclass-level value function online ~3 trillion state/action pairs x © 2004 – Carlos Guestrin, Daphne Koller instantiated Q-functions w. C in # agents grow polynomially Coordination graph computes argmaxa Q (x, a) World a

Demo: Generalized 9 x 3 Guestrin, Koller, Gearhart & Kanodia © 2004 – Carlos Guestrin, Daphne Koller

Tactical Generalization 3 v. 3 4 v. 4 Generalize n Planned in 3 Footmen versus 3 Enemies n Generalized to 4 Footmen versus 4 Enemies © 2004 – Carlos Guestrin, Daphne Koller

Demo: Planned Tactical 3 x 3 Guestrin, Koller, Gearhart & Kanodia © 2004 – Carlos Guestrin, Daphne Koller

Demo: Generalized Tactical 4 x 4 Guestrin, Koller, Gearhart & Kanodia [Guestrin, K. , Gearhart & Kanodia ‘ 03] © 2004 – Carlos Guestrin, Daphne Koller

Summary Distributed coordinated action selection Effective planning under uncertainty Generalization to new problems Structured Multi-Agent MDPs © 2004 – Carlos Guestrin, Daphne Koller

Important Questions Continuous spaces Complex actions Partial observability Learning to act How far can we go? ? © 2004 – Carlos Guestrin, Daphne Koller

http: //robotics. stanford. edu/~koller Carlos Guestrin Chris Gearhart Neal Kanodia Shobha Venkataraman Ronald Parr Curt Bererton Geoff Gordon Sebastian Thrun Jelle Kok Matthijs Spaan Nikos Vlassis