Policy Gradient Methods Sources Stanford CS 231 n

  • Slides: 25
Download presentation
Policy Gradient Methods Sources: Stanford CS 231 n, Berkeley Deep RL course, David Silver’s

Policy Gradient Methods Sources: Stanford CS 231 n, Berkeley Deep RL course, David Silver’s RL course Image source

Policy Gradient Methods • Instead of indirectly representing the policy using Q-values, it can

Policy Gradient Methods • Instead of indirectly representing the policy using Q-values, it can be more efficient to parameterize and learn it directly • Especially in large or continuous action spaces Image source: Open. AI Gym

Stochastic policy representation

Stochastic policy representation

Stochastic policy representation ? ? Source: D. Silver The agent can’t tell the difference

Stochastic policy representation ? ? Source: D. Silver The agent can’t tell the difference between the gray cells

Stochastic policy representation

Stochastic policy representation

Expected value of a policy

Expected value of a policy

Finding the policy gradient

Finding the policy gradient

Finding the policy gradient The score function

Finding the policy gradient The score function

Finding the policy gradient Gradient of log-likelihood of actions under current policy

Finding the policy gradient Gradient of log-likelihood of actions under current policy

Finding the policy gradient

Finding the policy gradient

REINFORCE algorithm Williams et al. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine

REINFORCE algorithm Williams et al. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3): 229 -256, 1992

REINFORCE: Single-step version

REINFORCE: Single-step version

Reducing variance

Reducing variance

Reducing variance

Reducing variance

Reducing variance

Reducing variance

Actor-Critic algorithm • Combine policy gradients and Q-learning by simultaneously training an actor (the

Actor-Critic algorithm • Combine policy gradients and Q-learning by simultaneously training an actor (the policy) and a critic (the Q-function) Source: D. Silver

Reducing variance Advantage function

Reducing variance Advantage function

Estimating the advantage function

Estimating the advantage function

Online actor-critic algorithm Source: Berkeley RL course

Online actor-critic algorithm Source: Berkeley RL course

Asynchronous advantage actor-critic (A 3 C) V, π Agent 1 Experience 1 Local updates

Asynchronous advantage actor-critic (A 3 C) V, π Agent 1 Experience 1 Local updates Agent 2 Experience 2 Local updates Agent n Experience n Local updates . . . Asynchronously update global parameters Mnih et al. Asynchronous Methods for Deep Reinforcement Learning. ICML 2016

Asynchronous advantage actor-critic (A 3 C) Mean and median human-normalized scores over 57 Atari

Asynchronous advantage actor-critic (A 3 C) Mean and median human-normalized scores over 57 Atari games Mnih et al. Asynchronous Methods for Deep Reinforcement Learning. ICML 2016

Asynchronous advantage actor-critic (A 3 C) TORCS car racing simulation video Mnih et al.

Asynchronous advantage actor-critic (A 3 C) TORCS car racing simulation video Mnih et al. Asynchronous Methods for Deep Reinforcement Learning. ICML 2016

Asynchronous advantage actor-critic (A 3 C) Motor control tasks video Mnih et al. Asynchronous

Asynchronous advantage actor-critic (A 3 C) Motor control tasks video Mnih et al. Asynchronous Methods for Deep Reinforcement Learning. ICML 2016

Benchmarks and environments for Deep RL Open. AI Gym https: //gym. openai. com/

Benchmarks and environments for Deep RL Open. AI Gym https: //gym. openai. com/

Policy Gradient Methods Sources Stanford CS 231 nPolicy Gradient Methods Sources Stanford CS 231 n
Gradient Methods Steepest Descent gradient descent Conjugate GradientGradient Methods Steepest Descent gradient descent Conjugate Gradient
C Watershed Implementation Slide 1 Gradient The gradientC Watershed Implementation Slide 1 Gradient The gradient
Thermoregulation Temperature Gradient for Kingsnake Temperature Gradient forThermoregulation Temperature Gradient for Kingsnake Temperature Gradient for
Second Order Closure Material Derivative Gradient terms GradientSecond Order Closure Material Derivative Gradient terms Gradient
CLIC MAIN PARAMETERS Accelerating Gradient What gradient couldCLIC MAIN PARAMETERS Accelerating Gradient What gradient could
Gradient Type 1 Gradient stops 5 Stop 1Gradient Type 1 Gradient stops 5 Stop 1
Gradient dune fonction Gnralits La notion de gradientGradient dune fonction Gnralits La notion de gradient
gradient Descent py def gradient Descentx y thetagradient Descent py def gradient Descentx y theta
Gradient Descent Review Gradient Descent In step 3Gradient Descent Review Gradient Descent In step 3
Training and optimization Stochastic gradient descent Gradient onTraining and optimization Stochastic gradient descent Gradient on
DTI Diffusion Sensitizing Gradient diffusion sensitizing gradient PlanningDTI Diffusion Sensitizing Gradient diffusion sensitizing gradient Planning
GRADIENT Himalayan elevation gradient NICHE temperature niches RichnessGRADIENT Himalayan elevation gradient NICHE temperature niches Richness
Conjugate Gradient Method for Indefinite Matrices Conjugate GradientConjugate Gradient Method for Indefinite Matrices Conjugate Gradient
C Watershed Implementation Slide 1 Gradient The gradientC Watershed Implementation Slide 1 Gradient The gradient
Second Order Closure Material Derivative Gradient terms GradientSecond Order Closure Material Derivative Gradient terms Gradient
Gradient of a line The gradient measures theGradient of a line The gradient measures the
GRADIENT Simple Gradient with Pythagoras Theorem Exam TypeGRADIENT Simple Gradient with Pythagoras Theorem Exam Type
Gradient Adiabatique Gradient adiabatique sec Dans la troposphreGradient Adiabatique Gradient adiabatique sec Dans la troposphre
Aortic Valve Stenosis High Gradient Vs Low GradientAortic Valve Stenosis High Gradient Vs Low Gradient
The Gradient The gradient m of a straightThe Gradient The gradient m of a straight
Finding the gradient In general the gradient ofFinding the gradient In general the gradient of
Numerical Methods Multidimensional Gradient Methods in Optimization TheoryNumerical Methods Multidimensional Gradient Methods in Optimization Theory
Numerical Methods Multidimensional Gradient Methods in Optimization TheoryNumerical Methods Multidimensional Gradient Methods in Optimization Theory