Policy Gradient in Continuous Time by Remi Munos

Policy Gradient in Continuous Time by Remi Munos, JMLR 2006 Presented by Hui Li Duke University Machine Learning Group May 30, 2007

Outline • Introduction • Discretized Stochastic Processes Approximation • Model-free Reinforcement Learning (RL) algorithm • Example Results

Introduction of the Problem • Consider an optimal control problem with continuous state System dynamics: Control State Ø Deterministic process Ø Continuous state • Objective: Find an optimal control (ut) that maximize the functional Objective function:

Introduction of the Problem • Consider a class of parameterized policies with • Find parameter that maximize the performance measure • Standard approach is to use gradient ascent method object of the paper

Introduction of the Problem How to compute • Finite-difference method This method requires a large number of trajectories to compute the gradient of performance measure. • Pathwise estimation of the gradient Compute the gradient using one trajectory only

Introduction of the Problem Pathwise estimation of the gradient Ø Define Ø Dynamics of zt: Ø Gradient unknown • In the reinforcement learning, to approximate zt? is unknown. How

Discretized Stochastic Processes Approximation • A General Convergence Result If

• Discretization of the state Ø Stochastic policy Ø Stochastic discrete state process Initialization: Jump in state

Proof of proposition 5: From Taylor’s formula The average jump: Directly apply the Theorem 3, proposition 5 is proved.

• Discretization of the state gradient Ø Stochastic discrete state gradient process Initialization: With

Proof of proposition 6: Since then Directly apply the Theorem 3, proposition 6 is proved.

Model-free Reinforcement Learning Algorithm Let In this stochastic approximation, is observed, and is given, we only need to approximate

Least-Square Approximation of Define The set of past discrete times t-c s t when action ut have been taken. From Taylor’s formula, for all discrete time s, We deduce

Where We may derive an approximation of the least-square problem: by solving Then we have Here denote the average value of

Algorithm

Experimental Results Six continuous state: x 0, y 0: hand position x, y: mass position vx, vy: mass velocity Four control action: U ={(1, 0), (0, 1), (1, 0), (0, -1)} Goal: reach a target (x. G, y. G) with the mass at specific time T Terminal reward function

The system dynamics: Consider a Boltzmann-like stochastic policy where

Conclusion • Described a reinforcement learning method for approximating the gradient of a continuous-time deterministic problem with respect to the control parameters • Used a stochastic policy to approximate the continuous system by a consistent stochastic discrete process