Batch Reinforcement Learning Alan Fern Based in part

Batch Reinforcement Learning Alan Fern * Based in part on slides by Ronald Parr

Overview h What is batch reinforcement learning? h Least Squares Policy Iteration h Fitted Q-iteration h Batch DQN

Online versus Batch RL h Online RL: integrates data collection and optimization 5 Select actions in environment and at the same time update parameters based on each observed (s, a, s’, r) h Batch RL: decouples data collection and optimization 5 First generate/collect experience in the environment giving a data set of state-action-reward-state pairs {(si, ai, ri, si’)} 5 We may not even know where the data came from 5 Use the fixed set of experience to optimize/learn a policy h Online vs. Batch: 5 Batch algorithms are often more “data efficient” and stable 5 Batch algorithms ignore the exploration-exploitation problem, and do their best with the data they have

Batch RL Motivation h There are many applications that naturally fit the batch RL model h Medical Treatment Optimization: 5 Input: collection of treatment episodes for an ailment giving sequence of observations and actions including outcomes 5 Ouput: a treatment policy, ideally better than current practice h Emergency Response Optimization: 5 Input: collection of emergency response episodes giving movement of emergency resources before, during, and after 911 calls 5 Output: emergency response policy

Batch RL Motivation h Online Education Optimization: 5 Input: collection of episodes of students interacting with an educational system that gives information and questions in order to teach a topic g Actions correspond to giving the student some information or giving them a question of a particular difficulty and topic 5 Ouput: a teaching policy that is tuned to student based on what is known about the student

Least Squares Policy Iteration (LSPI) h LSPI is a model-free batch RL algorithm 5 Learns a linear approximation of Q-function 5 stable and efficient 5 Never diverges or gives meaningless answers h LSPI can be applied to a dataset regardless of how it was collected 5 But garbage in, garbage out. Least-Squares Policy Iteration, Michail Lagoudakis and Ronald Parr, Journal of Machine Learning Research (JMLR), Vol. 4, 2003, pp. 1107 -1149.

Least Squares Policy iteration h No time to cover details of derivation 5 Details are in the appendix of these slides h LSPI is a wrapper around an algorithm LSTDQ h LSTDQ: learns a Q-function for current policy given the batch of data 5 Can learn Q-function for policy from any (reasonable) set of samples---sometimes called an off-policy method 5 No need to collect samples from current policy h Disconnects policy evaluation from data collection 5 Permits reuse of data across iterations! 5 Truly a batch method.

Implementing LSTDQ h

Running LSPI h There is a Matlab implementation available! 1. Collect a database of (s, a, r, s’) experiences (this is the magic step) 2. Start with random weights (= random policy) 3. Repeat 5 Evaluate current policy against database g Run LSTDQ to generate new set of weights g New weights imply new Q-function and hence new policy 5 Replace current weights with new weights h Until convergence

Results: Bicycle Riding h Watch random controller operate simulated bike h Collect ~40, 000 (s, a, r, s’) samples h Pick 20 simple feature functions ( 5 actions) h Make 5 -10 passes over data (PI steps) h Reward was based on distance to goal + goal achievement h Result: Controller that balances and rides to goal

Bicycle Trajectories

What about Q-learning? h Ran Q-learning with same features h Used experience replay for data efficiency

Q-learning Results

LSPI Robustness

Some key points h LSPI is a batch RL algorithm 5 Can generate trajectory data anyway you want 5 Induces a policy based on global optimization over full dataset h Very stable with no parameters that need tweaking

So, what’s the bad news? h LSPI does not address the exploration problem 5 It decouples data collection from policy optimization 5 This is often not a major issue, but can be in some cases h k 2 can sometimes be big 5 Lots of storage 5 Matrix inversion can be expensive h Bicycle needed “shaping” rewards h Still haven’t solved 5 Feature selection (issue for all machine learning, but RL seems even more sensitive)

Fitted Q-Iteration h LSPI is limited to linear functions over a given set of features h Fitted Q-Iteration allows us to use any type of function approximator for the Q-function 5 Random Forests have been popular 5 Deep Networks h Fitted Q-Iteration is a very straightforward batch version of Q-learning Damien Ernst, Pierre Geurts, Louis Wehenkel. (2005). Tree-Based Batch Mode Reinforcement Learning Journal of Machine Learning Research; 6(Apr): 503— 556.

Fitted Q-Iteration h

DQN h DQN was developed by Deep. Mind originally for online learning of Atari games h However, the algorithm can be used effectively as is for Batch RL. h I haven’t seen this done, but it is straightforward.

DQN for Batch RL h

Appendix 21

Projection Approach to Approximation h Recall the standard Bellman equation: or equivalently Bellman operator where T[. ] is the h Recall from value iteration, the sub-optimality of a value function can be bounded in terms of the Bellman error: h This motivates trying to find an approximate value function with small Bellman error

Projection Approach to Approximation h Suppose that we have a space of representable value functions 5 E. g. the space of linear functions over given features h Let P be a projection operator for that space 5 Projects any value function (in or outside of the space) to “closest” value function in the space h “Fixed Point” Bellman Equation with approximation 5 Depending on space this will have a small Bellman error h LSPI will attempt to arrive at such a value function 5 Assumes linear approximation and least-squares projection

Projected Value Iteration h Naïve Idea: try computing projected fixed point using VI h Exact VI: (iterate Bellman backups) h Projected VI: (iterated projected Bellman backups): Projects exact Bellman backup to closest function in our restricted function space exact Bellman backup (produced value function)

Example: Projected Bellman Backup Restrict space to linear functions over a single feature f: Suppose just two states s 1 and s 2 with: f(s 1)=1, f(s 2)=2 Suppose exact backup of Vi gives: Can we represent this exact backup in our linear space? No f(s 1)=1 f(s 2)=2

Example: Projected Bellman Backup Restrict space to linear functions over a single feature f: Suppose just two states s 1 and s 2 with: f(s 1)=1, f(s 2)=2 Suppose exact backup of Vi gives: The backup can’t be represented via our linear function: projected backup is just least-squares fit to exact backup f(s 1)=1 f(s 2)=2

Problem: Stability h Exact value iteration stability ensured by contraction property of Bellman backups: h Is the “projected” Bellman backup a contraction: ?

Example: Stability Problem [Bertsekas & Tsitsiklis 1996] Problem: Most projections lead to backups that are not contractions and unstable s 1 s 2 Rewards all zero, single action, g = 0. 9: V* = 0 Consider linear approx. w/ single feature f with weight w. Optimal w = 0 since V*=0

Example: Stability Problem f(s 1)=1 Vi(s 1) = wi s 1 s 2 f(s 2)=2 weight value at iteration i Vi(s 2) = 2 wi From Vi perform projected backup for each state Can’t be represented in our space so find wi+1 that gives least-squares approx. to exact backup After some math we can get: wi+1 = 1. 2 wi What does this mean?

Example: Stability Problem V(x) Iteration # 1 2 S Each iteration of Bellman backup makes approximation worse! Even for this trivial problem “projected” VI diverges.

Understanding the Problem h What went wrong? 5 Exact Bellman backups reduces error in max-norm 5 Least squares (= projection) non-expansive in L 2 norm g But may increase max-norm distance! h Conclusion: Alternating Bellman backups and projection is risky business

OK, What’s LSTD? h

Deriving LSTD assigns a value to every state is a linear function in the column space of f 1…fk, that is, K basis functions f 1(s 1) f 2(s 1). . . f 1(s 2) f 2(s 2)… F= . . . # states

Suppose we know true value of policy h We would like the following: h Least squares weights minimizes squared error Sometimes called pseudoinverse h Least squares projection is then Textbook least squares projection operator

But we don’t know V… h Recall fixed-point equation for policies h Will solve a projected fixed-point equation: h Substituting least squares projection into this gives: h Solving for w:

Almost there… h Matrix to invert is only K x K h But… 5 Expensive to construct matrix (e. g. P is |S|x|S|) g Presumably we are using LSPI because |S| is enormous 5 We don’t know P 5 We don’t know R

Using Samples for F Suppose we have state transition samples of the policy running in the MDP: {(si, ai, ri, si’)} Idea: Replace enumeration of states with sampled states K basis functions f 1(s 1) f 2(s 1). . . f 1(s 2) f 2(s 2)…. . . samples states

Using Samples for R Suppose we have state transition samples of the policy running in the MDP: {(si, ai, ri, si’)} Idea: Replace enumeration of reward with sampled rewards r 1 r 2 R = . . . samples

40

Using Samples for PF Idea: Replace expectation over next states with sampled next states. K basis functions f 1(s 1’) f 2(s 1’). . . f 1(s 2’) f 2(s 2’)…. . . s’ from (s, a, r, s’)

Putting it Together h LSTD needs to compute: from previous slide h The hard part of which is B the kxk matrix: h Both B and b can be computed incrementally for each (s, a, r, s’) sample: (initialize to zero)

LSTD Algorithm h Collect data by executing trajectories of current policy h For each (s, a, r, s’) sample:

LSTD Summary h Does O(k 2) work per datum 5 Linear in amount of data. h Approaches model-based answer in limit h Finding fixed point requires inverting matrix h Fixed point almost always exists h Stable; efficient

Approximate Policy Iteration with LSTD Policy Iteration: iterates between policy improvement and policy evaluation Idea: use LSTD for approximate policy evaluation in PI Start with random weights w (i. e. value function) Repeat Until Convergence = greedy( ) // policy improvement Evaluate using LSTD g Generate sample trajectories of g Use LSTD to produce new weights w (w gives an approx. value function of )

What Breaks? h No way to execute greedy policy without a model h Approximation is biased by current policy 5 We only approximate values of states we see when executing the current policy 5 LSTD is a weighted approximation toward those states h Can result in Learn-forget cycle of policy iteration 5 Drive off the road; learn that it’s bad 5 New policy never does this; forgets that it’s bad h Not truly a batch method 5 Data must be collected from current policy for LSTD

LSPI h LSPI is similar to previous loop but replaces LSTD with a new algorithm LSTDQ h LSTD: produces a value function 5 Requires samples from policy under consideration h LSTDQ: produces a Q-function for current policy 5 Can learn Q-function for policy from any (reasonable) set of samples---sometimes called an off-policy method 5 No need to collect samples from current policy h Disconnects policy evaluation from data collection 5 Permits reuse of data across iterations! 5 Truly a batch method.