Competition and cooperation between reinforcement learning systems Sam

Competition and cooperation between reinforcement learning systems Sam Gershman Department of Psychology Harvard University

Roadmap 1. The existence of multiple reinforcement learning systems 2. Competitive architectures 3. Cooperative architectures

Multiple learning systems

Multiple learning systems “Model-based” system builds a cognitive map of the environment and uses it to plan

Multiple learning systems “Model-based” system builds a cognitive map of the environment and uses it to plan Left Right S 1 0. 5 0. 8 S 2 1 0 S 3 1 1 “Model-free” system learns to predict future reward from trial and error

Multiple learning systems “Model-based” system builds a cognitive map of the environment and uses it to plan Flexible Inefficient Left Right S 1 0. 5 0. 8 S 2 1 0 S 3 1 1 “Model-free” system learns to predict future reward from trial and error Inflexible Efficient

Model-free learning and control Model-free values are updated using dopaminergic prediction errors. RPE = Observed – Expected Law of effect: previously rewarded actions are more likely to be chosen again in the future. Schultz (1998)

Model-based learning and control • Value iteration (dynamic programming): use Bellman equation to iteratively refine value function – Global (estimates all values) – Inefficient (curse of dimensionality) • Tree search: forward simulate from current state – Local (estimates small subset of values) – Much more efficient (can win Go!) • We don’t currently know much about the detailed algorithmic structure of model-based learning or control.

Architectures Competitive • Uncertainty-based arbitration: choose controller with more accurate value estimates • Cost-benefit arbitration: choose controller that is likely to optimize reward relative to cognitive effort

Architectures Competitive Cooperative • Uncertainty-based arbitration: choose controller with more accurate value estimates • Cost-benefit arbitration: choose controller that is likely to optimize reward relative to cognitive effort • Dyna: model-based simulation to train modelfree learning • Plan-until-habit: truncate model-based planning with model-free values • Heuristic search: use model -free values to guide tree search • Habitual goal selection: model-free controller chooses goals for modelbased system to pursue

Evidence for competition Moderate training: sensitive to devaluation (model-based) Extensive training: insensitive to devaluation (model-free)

57% % 26% 70 70% Learning in the human brain 41% 28% (all slowly changing) Daw, Gershman et al. (2011)

What happens after a rare transition results in reward? 30%

Predictions model-free ignores transition structure model-based respects transition structure

Behavior model-based model-free Behavior is a mixture of model-based and model -free learning.

Striatal activity reflects mixture

Clinical implications Relative reliance on model-based correlates with several clinical pathologies. Gillan et al. (2016)

Developmental implications Increasing reliance on model-based control over the course of development. Decker et al. (2016)

Cognitive load makes people rely on the model-free system Otto, Gershman et al. (2013)

Principles of competition • Uncertainty-based arbitration posits that controllers are selected on the basis of their variances. – This can’t explain incentive effects. • Cost-benefit analysis: when model-based control is too cognitively costly relative to its accuracy (e. g. , under working memory load), then switch to model-free control.

Being more model-based in the Daw task does not increase reward rate Simulation Data Kool, Cushman & Gershman (2016)

New task Reward rate Simulation Larger drift rate Points instead of probabilities New structure 1 second stage choice Model-based control Kool, Cushman & Gershman (2016)

New task Data Larger drift rate Points instead of probabilities New structure 1 second stage choice Kool, Cushman & Gershman (2016)

Reinforcement learning as costbenefit decision making? ?

Stake manipulation Reward rate Simulation Multipliers conditions: 50% w

If planning is costly, then increase W on vs trials

(new Kool task) (original Daw task)

Cooperative architectures • Dyna: model-based simulation to train modelfree learning • Plan-until-habit: truncate model-based planning with model-free values • Heuristic search: use model -free values to guide tree search • Habitual goal selection: model-free controller chooses goals for modelbased system to pursue

The Dyna architecture Sutton (1990) proposed a cooperative architecture called Dyna

The Dyna architecture Sutton (1990) proposed a cooperative architecture called Dyna The model-free system controls behavior

The Dyna architecture Sutton (1990) proposed a cooperative architecture called Dyna The model-based system learns the transition and reward functions

The Dyna architecture Sutton (1990) proposed a cooperative architecture called Dyna Offline, the model-based system simulates experience from the model

The Dyna architecture Sutton (1990) proposed a cooperative architecture called Dyna Simulated experience is used to train the modelfree system

Key findings • Giving people a quiet rest period (opportunity for offline simulation) changes their subsequent decision making (Gershman, Markman & Otto, 2014). • This effect is invariant to cognitive load in the final decision phase, suggesting that decisions are not under model-based control.

Cooperative architectures • Dyna: model-based simulation to train modelfree learning • Plan-until-habit: truncate model-based planning with model-free values • Heuristic search: use model-free values to guide tree search • Habitual goal selection: model-free controller chooses goals for modelbased system to pursue

Experts don’t search much more deeply than non-experts. Definitely much less than the enormous search trees constructed by computer programs like Deep. Blue. The difference between experts and non-experts is not in how deep they search, but how they choose which paths to search.

Pattern recognition • Chase and Simon studied memory in chess players as a window into their pattern recognition abilities. • They hypothesized that experts store large libraries of “chunks” that allow them to quickly evaluate positions. • In RL terminology, we can think of this pattern recognition mechanism as a model-free estimate of the value function, which can be used to guide model-based search.

Summary • Model-based and model-free systems compete for control of behavior. • Evidence for cost-benefit analysis in arbitration between the systems. • Systems can also cooperate in several different ways. • Competition and cooperation aren’t mutually exclusive.

Future directions • Other controllers: episodic and Pavlovian. • Better process models of planning. • Rigorous economic analysis: how can we quantify and measure the cognitive cost of model -based control?

Extra slides

Metacontrol model Alternative interpretation: stochastically choose model-based system with probability w. How should we choose w?

Policy gradient algorithm Choose weight that maximizes value. +αRPE if model-based - αRPE if model-free Δw = Increase probability of using a system when it gets a positive prediction error, decrease probability when it gets a negative prediction error.

Puzzle: why are model-based and model-free signals multiplexed? Model-free prediction error Model-based influence Meta-control policy implies that prediction errors will sometimes between model-free, sometimes model-based, depending on which system is in control.

Another puzzle: dopamine Dopamine makes people more model-based. Wunderlich et al. (2012) Sharp et al. (2016)

Dopamine and effort “liking” vs. “wanting” Dopamine depletion eliminates willingness to work for a reward. By extension, increasing dopamine will increase willingness to expend cognitive effort for reward.