MonteCarlo Planning Policy Improvement Alan Fern 1 MonteCarlo

Monte-Carlo Planning: Policy Improvement Alan Fern 1

Monte-Carlo Planning h Often a simulator of a planning domain is available or can be learned from data Conservation Planning Fire & Emergency Response 2

Large Worlds: Monte-Carlo Approach h Often a simulator of a planning domain is available or can be learned from data h Monte-Carlo Planning: compute a good policy for an MDP by interacting with an MDP simulator World Simulator action Real World State + reward 3

MDP: Simulation-Based Representation h A simulation-based representation gives: S, A, R, T, I: 5 finite state set S (|S|=n and is generally very large) 5 finite action set A (|A|=m and will assume is of reasonable size) 5 Stochastic, real-valued, bounded reward function R(s, a) = r g Stochastically returns a reward r given input s and a 5 Stochastic transition function T(s, a) = s’ (i. e. a simulator) g g Stochastically returns a state s’ given input s and a Probability of returning s’ is dictated by Pr(s’ | s, a) of MDP 5 Stochastic initial state function I. g Stochastically returns a state according to an initial state distribution These stochastic functions can be implemented in any language! 4

Outline h You already learned how to evaluate a policy given a simulator 5 Just run the policy multiple times for a finite horizon and average the rewards h In next two lectures we’ll learn how to use the simulator in order to select good actions in the real world 5

Monte-Carlo Planning Outline h Single State Case (multi-armed bandits) 5 A basic tool for other algorithms h Monte-Carlo Policy Improvement 5 Policy rollout 5 Policy Switching Today h Monte-Carlo Tree Search 5 Sparse Sampling 5 UCT and variants 6

Single State Monte-Carlo Planning h Suppose MDP has a single state and k actions 5 Can sample rewards of actions using calls to simulator 5 Sampling action a is like pulling slot machine arm with random payoff function R(s, a) s a 1 a 2 ak … R(s, a 1) R(s, a 2) … R(s, ak) Multi-Armed Bandit Problem 7

Multi-Armed Bandits h We will use bandit algorithms as components for multi-state Monte-Carlo planning 5 But they are useful in their own right h Pure bandit problems arise in many applications h Applicable whenever: 5 We have a set of independent options with unknown utilities 5 There is a cost for sampling options or a limit on total samples 5 Want to find the best option or maximize utility of our samples

Multi-Armed Bandits: Examples h Clinical Trials 5 Arms = possible treatments 5 Arm Pulls = application of treatment to inidividual 5 Rewards = outcome of treatment 5 Objective = determine best treatment quickly h Online Advertising 5 Arms = different ads/ad-types for a web page 5 Arm Pulls = displaying an ad upon a page access 5 Rewards = click through 5 Objective = find best add quickly (the maximize clicks)

Simple Regret Objective h Different applications suggest different types of bandit objectives. h Today minimizing simple regret will be the objective 5 Simple Regret Minimization (informal): quickly identify arm with close to optimal expected reward s a 1 a 2 ak … R(s, a 1) R(s, a 2) … R(s, ak) Multi-Armed Bandit Problem 10

Simple Regret Objective: Formal Definition 11

Uniform. Bandit Algorith (or Round Robin) Bubeck, S. , Munos, R. , & Stoltz, G. (2011). Pure exploration in finitely-armed and continuous-armed bandits. Theoretical Computer Science, 412(19), 1832 -1852 h 12

Can we do better? Tolpin, D. & Shimony, S, E. (2012). MCTS Based on Simple Regret. AAAI Conference on Artificial Intelligence. h Often is more effective than Uniform. Bandit in practice. 13

Monte-Carlo Planning Outline h Single State Case (multi-armed bandits) 5 A basic tool for other algorithms h Monte-Carlo Policy Improvement 5 Policy rollout 5 Policy Switching Today h Monte-Carlo Tree Search 5 Sparse Sampling 5 UCT and variants 14

Policy Improvement via Monte-Carlo h Now consider a very large multi-state MDP. h Suppose we have a simulator and a non-optimal policy 5 E. g. policy could be a standard heuristic or based on intuition h Can we somehow compute an improved policy? World Simulator + Base Policy action Real World State + reward 15

Policy Improvement Theorem h 16

Policy Improvement via Bandits s a 1 ak a 2 … Sim. Q(s, a 1, π, h) Sim. Q(s, a 2, π, h) Sim. Q(s, ak, π, h) How to implement Sim. Q? 17

Policy Improvement via Bandits Sim. Q(s, a, π, h) q = R(s, a) simulate a in s s = T(s, a) for i = 1 to h-1 q = q + R(s, π(s)) simulate h-1 steps s = T(s, π(s)) of policy Return q Trajectory under p … Sum of rewards = Sim. Q(s, a 1, π, h) a 2 … Sum of rewards = Sim. Q(s, a 2, π, h) … Sum of rewards = Sim. Q(s, ak, π, h) ak … s a 1 18

Policy Improvement via Bandits Sim. Q(s, a, π, h) q = R(s, a) simulate a in s s = T(s, a) for i = 1 to h-1 q = q + R(s, π(s)) simulate h-1 steps s = T(s, π(s)) of policy Return q h Simply simulate taking a in s and following policy for h-1 steps, returning discounted sum of rewards h Expected value of Sim. Q(s, a, π, h) is Qπ(s, a, h) Ø So averaging across multiple runs of Sim. Q quickly converges to Qπ(s, a, h) 19

Policy Improvement via Bandits s a 1 ak a 2 … Sim. Q(s, a 1, π, h) Sim. Q(s, a 2, π, h) Sim. Q(s, ak, π, h) 20

Uniform. Rollout h s a 1 ak a 2 … Sim. Q(s, ai, π, h) trajectories … q 21 q 22 … q 2 w … … … q 11 q 12 … q 1 w … … … Samples of Sim. Q(s, ai, π, h) … … Each simulates taking action ai then following π for h-1 steps. qk 1 qk 2 … qkw 21

h Allocates a non-uniform number of trials across actions (focuses on more promising s actions) a 1 ak a 2 … … … q 11 q 12 … q 1 u q 21 q 22 … q 2 v qk 1 22

Executing Rollout in Real World a 2 Real world state/action s sequence ak … … run policy rollout a 1 Simulated experience ak a 2 run policy rollout a 1 ak a 2 … … … … … How much time does each decision take? 23

Policy Rollout: # of Simulator Calls s a 1 ak a 2 … Sim. Q(s, ai, π, h) trajectories … … … … … Each simulates taking action ai then following π for h-1 steps. • Total of n Sim. Q calls each using h calls to simulator and policy • Total of hn calls to the simulator and to the policy (dominates time to make decision) 24

Practical Issues: Accuracy h In general, larger values are better, but this increases time. 25

Practical Issues: Speed h There are three ways to speedup decision making time 1. Use a faster policy 26

Practical Issues: Speed h There are three ways to speedup decision making time 1. Use a faster policy 2. Decrease the number of trajectories n h Decreasing Trajectories: 5 If n is small compared to # of actions k, then performance could be poor since actions don’t get tried very often 5 One way to get away with a smaller n is to use an action filter h Action Filter: a function f(s) that returns a subset of the actions in state s that rollout should consider 5 You can use your domain knowledge to filter out obviously bad actions 5 Rollout decides among the remaining actions returned by f(s) 5 Since rollout only tries actions in f(s) can use a smaller value of n 27

Practical Issues: Speed h There are three ways to speedup either rollout procedure 1. Use a faster policy 2. Decrease the number of trajectories n 3. Decrease the horizon h h Decrease Horizon h: 5 If h is too small compared to the “real horizon” of the problem, then the Q-estimates may not be accurate 5 Can get away with a smaller h by using a value estimation heuristic h Heuristic function: a heuristic function v(s) returns an estimate of the value of state s 5 Sim. Q is adjusted to run policy for h steps ending in state s’ and returns the sum of rewards up until s’ added to the estimate v(s’) 28

Multi-Stage Rollout h A single call to Rollout[π, h, w](s) yields one iteration of policy improvement starting at policy π h We can use more computation time to yield multiple iterations of policy improvement via nesting calls to Rollout 5 Rollout[π, h, w](s) returns the action for state s resulting from two iterations of policy improvement 5 Can nest this arbitrarily h Gives a way to use more time in order to improve performance 29

Multi-Stage Rollout s Each step requires nh simulator calls for Rollout policy a 1 ak a 2 … Trajectories of Sim. Q(s, ai, Rollout[π, h, w], h) … … … … … • Two stage: compute rollout policy of “rollout policy of π” • Requires (nh)2 calls to the simulator for 2 stages • In general exponential in the number of stages 30

Example: Rollout for Solitaire [Yan et al. NIPS’ 04] Player Success Rate Time/Game Human Expert 36. 6% 20 min (naïve) Base Policy 1 rollout 13. 05% 0. 021 sec 31. 20% 0. 67 sec 2 rollout 47. 6% 7. 13 sec 3 rollout 56. 83% 1. 5 min 4 rollout 60. 51% 18 min 5 rollout 70. 20% 1 hour 45 min h Multiple levels of rollout can payoff but is expensive 31

Rollout in 2 -Player Games h s a 1 ak a 2 … p 1 p 2 … q 21 q 22 … q 2 w … … … … q 11 q 12 … q 1 w qk 1 qk 2 … qkw 32

Another Useful Technique: Policy Switching h Suppose you have a set of base policies {π1, π2, …, πM} h Also suppose that the best policy to use can depend on the specific state of the system and we don’t know how to select. h Policy switching is a simple way to select which policy to use at a given step via a simulator 33

Another Useful Technique: Policy Switching s π1 π2 πM … Sim(s, π1, h) Sim(s, π2, h) Sim(s, πM, h) h The stochastic function Sim(s, π, h) simply samples the h-horizon value of π starting in state s h Implement by simply simulating π starting in s for h steps and returning discounted total reward h Use Bandit algorithm to select best policy and then select action chosen by that policy 34

Policy. Switching Policy. Switch[{π1, π2, …, πM}, h, n](s) 1. Define bandit with M arms giving rewards Sim(s, πi, h) 2. Let i* be index of the arm/policy selected by your favorite bandit algorithm using n trials 3. Return action πi*(s) π1 s πM π2 … Sim(s, πi, h) trajectories Each simulates following πi for h steps. … v 21 v 22 … v 2 w … … … v 11 v 12 … v 1 w … … … Discounted cumulative rewards v. M 1 v. M 2 … v. Mw 35

Executing Policy Switching in Real World �� 2(s) Real world state/action s sequence �� k(s’) … … run policy rollout �� �� k 2 Simulated experience run policy rollout �� 1 �� �� k 2 … … … … … 36

Policy Switching: Quality h 37

Policy Switching in 2 -Player Games

Maxi. Min Policy Switching …. Current State s Build Game Matrix …. …. …. …. …. Game Simulator Each entry gives estimated value (for max player) of playing a policy pair against one another Each value estimated by averaging across w simulated games.

Maxi. Min Switching …. Current State s Build Game Matrix …. …. …. Game Simulator Can switch between policies based on state of game! ….

Maxi. Min Switching …. Current State s Build Game Matrix …. …. …. Game Simulator ….

Policy Switching: Quality h Maxi. Min policy switching will often do better than any single policy in practice h The theoretical guarantees for basic Maxi. Min policy switching are quite weak 5 Tweaks to the algorithm can fix this h For single-agent MDPs, policy switching is guaranteed to improve over the best policy in the set. 42