Using Inaccurate Models in Reinforcement Learning S T

Using Inaccurate Models in Reinforcement Learning S T A N F O R D Pieter Abbeel, Morgan Quigley, and Andrew Y. Ng 1. Preliminaries 0. Overview n RL for high-dimensional continuous state-space tasks. n Markov Decision Process (MDP) n n We present a hybrid algorithm, which requires only Student-driver learning to make a 90 degree right turn: Only a few trials needed, no accurate model. n n Time varying, deterministic dynamics Key aspects Real-life trial: shows whether turn is wide or short. n Crude model: turning steering wheel more to the right results in sharper turn; turning steering wheel more to the left results in wider turn. n T = { ft : S x A ! S, t = 0, …, H}. n Goal: find policy : S ! A, that H maximizes U( ) = E [ R(st) | ]. t=0 n an approximate model, n a small number of real-life trials. n n 2. Motivating Example M = (S, A, T , H, s 0, R). n n S = (continuous state space) Model-based RL: Difficult to build an accurate model. n Model-free RL: Often requires large numbers of real-life trials. n n S T A N F O R D Resulting policy is an approximate local optimum. n Focus: task of trajectory following. n Result: good policy gradient estimate. 4. Main Idea Effect on Policy Gradient Estimate Test the model-based optimal policy in real-life. Solution: How to proceed when the real-life trajectory is not the desired trajectory predicted by the model? Update the model such that it becomes exact for the current policy. The policy gradient is zero according to the model, so no improvement is possible based on the model. More specifically, add a bias to the model for each time step. See illustration below for details. n Exact policy gradient: n Model based policy gradient: Real-life trajectory n Trajectory predicted by model (equals desired trajectory) Two sources of error: Derivative of approximate transition function The new model perfectly predicts the state sequence obtained by the current policy. Consequently, the new model “knows” that more right steering is required. 5. Complete Algorithm 7. Experiments 1. Find the (locally) optimal policy for the model. 2. Execute the current policy and record the state trajectory. 3. Update the model such that the new model is exact for the current policy . 4. Compute the policy gradient in the new model and update the policy: : = + . 5. Go back to Step 2. Control actions: throttle and steering. n We used DDP. n Our algorithm took 10 iterations. n Figure-8 Maneuver Open Loop Turn 76% utility improvement over model-based RL 6. Theoretical Guarantees n Real RC Car n Notes: n The step-size parameter is determined by a line search. n Instead of the policy gradient, any algorithm that provides a local policy improvement direction can be used. In our experiments we used differential dynamic programming. Our algorithm eliminates the second source of error. Videos available. Flight Simulator We generated “approximate models” by randomly perturbing the 43 model parameters. n All 4 standard fixed-wing control actions: throttle, ailerons, elevators and rudder. n We used differential dynamic programming (DDP) for the model -based RL and to provide local policy improvements. n Our algorithm took 5 iterations. Fixed-wing flight simulator available at: http: //sourceforge. net/projects/aviones. Evaluation of derivatives along wrong trajectory Improvements over modelbased RL: desired trajectory model-based controller our algorithm Circle Maneuver Turn: 97% Circle: 88% Figure-8: 67% Let the local policy improvement algorithm be policy gradient. 8. Related Work n Iterative Learning Control: Uchiyama (1978), Longman et al. (1992), Moore (1993), Horowitz (1993), Bien et al. (1991), Owens et al. (1995), Chen et al. (1997), … n n Successful robot control with limited number of trials: n Notes: n These assumptions are insufficient to give the same performance guarantees for model-based RL. n The constant K depends only on the dimensionality of the state, action, and policy ( ), the horizon H and an upper bound on the 1 st and 2 nd derivatives of the transition model, the policy and the reward function. n Non-parametric learning: n n Atkeson and Schaal (1997), Morimoto and Doya (2001). Atkeson et al. (1997). Classical and robust control theory: Anderson and Moore (1989), Zhou et al. (1995), Bagnell et al. (2001), Morimoto and Atkeson (2002), … n 9. Conclusion We presented an algorithm that uses a crude model and a small number of real-life trials to find a policy that works well in real-life. n Our theoretical results show that---assuming a deterministic setting and an approximate model---our algorithm returns a policy that is (locally) near-optimal. n Our experiments show that our algorithm can significantly improve on purely model-based RL by using only a small number of real-life trials, even when the true system is not deterministic. n