Brain Dis C Ph D Conference Active inference

Brain. Dis. C Ph. D Conference: Active inference and epistemic value Karl Friston, University College London Abstract: I will talk about a formal treatment of choice behaviour based on the premise that agents minimise the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (intrinsic) value. Minimising expected free energy is therefore equivalent to maximising extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximising information gain or intrinsic value; i. e. , reducing uncertainty about the causes of valuable outcomes. The resulting scheme resolves the exploration-exploitation dilemma: epistemic value is maximised until there is no further information gain, after which exploitation is assured through maximisation of extrinsic value. This is formally consistent with the Infomax principle, generalising formulations of active vision based upon salience (Bayesian surprise) and optimal decisions based on expected utility and risk sensitive (KL) control. Furthermore, as with previous active inference formulations of discrete (Markovian) problems; ad hoc softmax parameters become the expected (Bayesoptimal) precision of beliefs about – or confidence in – policies. We focus on the basic theory – illustrating the minimisation of expected free energy using simulations. A key aspect of this minimisation is the similarity of precision updates and dopaminergic discharges observed in conditioning paradigms. Key words: active inference ∙ cognitive ∙ dynamics ∙ free energy ∙ epistemic value ∙ self-organization

Does the brain use continuous or discrete state-space models?

Does the brain use continuous or discrete state-space models? States and their Sufficient statistics

Approximate Bayesian inference for continuous states: Bayesian filtering States and their Sufficient statistics Free energy Model evidence

“Objects are always imagined as being present in the field of vision as would have to be there in order to produce the same impression on the nervous mechanism” - von Helmholtz Hermann von Helmholtz Richard Gregory Impressions on the Markov blanket…

Bayesian filtering and predictive coding prediction error update

Making our own sensations – predictions Prediction error Action Perception Changing sensations Changing predictions

Hierarchical generative models what A simple hierarchy the Ascending prediction errors where Descending predictions Sensory fluctuation s

David Mumford Predictive coding with reflexes Action oculomot or signals reflex arc pons proprioceptive input retinal input frontal eye fields geniculate Top-down or backward predictions Bottom-up or forward prediction error Perception Prediction error (superficial pyramidal cells) Expectations (deep pyramidal cells) visual cortex

What does Bayesian filtering explain?

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P 300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P 300) responses v. Specify a deep (hierarchical) generative model Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) v. Simulate approximate (active) Bayesian inference by Biased competition solving: Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control prediction update Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

An example: Visual searches and saccades

Free energy Likelihood Empirical priors Prior beliefs Entropy Energy “I am [ergodic] therefore I think [I will minimise free energy]” Expected energy Extrinsic value Expected entropy Epistemic value or information gain Bayesian surprise and Infomax KL or risk-sensitive control Expected utility theory In the absence of prior beliefs about outcomes: In the absence of ambiguity: In the absence of uncertainty or risk: Bayesian surprise Extrinsic value Predicted divergence Predicted mutual information

Free energy Likelihood Empirical priors Prior beliefs Entropy Energy “I am [ergodic] therefore I think [I will minimise free energy]” Epistemic value or information gain Extrinsic value stimulus visual input salience Sampling the world to resolve uncertainty sampling

Deep (hierarchical) generative model Parietal (where) Frontal eye fields Visual cortex Pulvinar salience map Fusiform (what) oculomotor reflex arc Superior colliculus

Saccadic eye movements Saccadic fixation and salience maps Hidden (oculomotor) states Visual samples Conditional expectations about hidden (visual) states And corresponding percept vs.

Is Bayesian filtering the only process theory for approximate Bayesian inference in the brain? What about state space models?

A (Markovian) generative model Likelihood Control states Empirical priors – hidden states – control states Hidden states Full priors

Prior beliefs about policies Quality of a policy = (negative) expected free energy Extrinsic value Epistemic value or information gain Bayesian surprise and Infomax KL or risk-sensitive control Expected utility theory In the absence of prior beliefs about outcomes: In the absence of ambiguity: In the absence of uncertainty or risk: Bayesian surprise Extrinsic value Predicted divergence Predicted mutual information

Generative models Discrete states Continuous states Control states Hidden states Bayesian filtering (predictive coding) Variational Bayes (belief updating)

Variational updates Functional anatomy Perception Action selection motor Cortex Incentive salience striatum occipital Cortex prefrontal Cortex hippocampus Performance FE KL RL success rate (%) 80 DA 60 40 20 0 0 0. 2 0. 4 0. 6 0. 8 1 Prior preference Simulated behaviour VTA/SN

Variational updating Functional anatomy Perception Action selection motor Cortex Incentive salience striatum occipital Cortex prefrontal Cortex hippocampus 30 25 20 15 10 VTA/SN 5 0. 2 0. 4 0. 6 0. 8 1 1. 2 1. 4 0. 08 0. 06 0. 04 0. 02 0 -0. 02 0. 25 0. 75 1 1. 25 1. 5 0. 2 0. 15 0. 1 0. 05 0 -0. 05 10 20 30 40 50 60 70 80 90 Simulated neuronal behaviour

What does believe updating explain? Cross frequency coupling Perceptual categorisation Violation (P 300) responses v. Specify a deep (hierarchical) generative model Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) v. Simulate approximate (active) Bayesian inference by Biased competition solving: Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

An example: Exploration and exploitation

Generative model (A, B, C) Control states 1 2 3 location 4 US US Hidden states CS location Outcomes context

Functional anatomy Action (and Bayesian model averaging) State estimation (planning) Motor Cortex State estimation (habitual) Dorsal prefrontal Striatum Policy selection Hippocampus VTA/SN Precision Learning Variational updates Ventral prefrontal Occipital Cortex Cerebellum

Functional anatomy Predicted action Dorsal prefrontal Motor Cortex Bayesian model average of next state Policy selection Hippocampus Precision Ventral prefrontal Striatum & VTA/SN State estimation under plausible policies Evaluation of policies Occipital Cortex Sensory input Habit learning Cerebellum

What does Bayesian filtering explain? Local field potentials 0. 14 0. 12 0. 1 Response Cross frequency coupling (phase precession) Perceptual categorisation Violation (P 300) responses Oddball (MMN) responses Unit responses Omission responses Attentional cueing (Posner paradigm) Biased competition 8 Visual illusions Dissociative symptoms Sensory attenuation 16 Sensorimotor integration Optimal motor control 24 Motor gain control 0. 25 0. 5 time (seconds) Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics 0. 08 0. 06 0. 04 0. 02 0 -0. 02 0. 75 0. 25 0. 5 time (seconds) 0. 75

What does Bayesian filtering explain? change in precision Response frequency Time-frequency (and phase) response Cross frequency coupling (phase synchronisation) 30 Perceptual categorisation 25 Violation (P 300) responses 20 Oddball (MMN) responses 15 Omission responses 10 5 Attentional cueing (Posner paradigm) 1 2 3 4 5 6 time (seconds) Biased competition Local field potentials Visual illusions Dissociative symptoms 0. 4 Sensory attenuation 0. 2 Sensorimotor integration 0 Optimal motor control -0. 2 Motor gain control 0. 250. 75 1 1. 251. 75 2 2. 252. 75 3 3. 253. 75 4 4. 254. 75 5 5. 255. 75 6 time (seconds) Oculomotor control and smooth pursuit Phasic dopamine responses Visual searches and saccades Action observation and mirror neuron responses 0. 15 Dopamine and affordance 0. 1 Action sequences (reversal learning) 0. 05 Interoceptive inference 0 Communication and hermeneutics 50 100 150 200 250 300 350 time (updates)

What does Bayesian filtering explain? Time-frequency (and phase) response frequency Cross frequency coupling 30 Perceptual categorisation 25 Violation (P 300) responses 20 Oddball (MMN) responses 15 10 Omission responses 5 Attentional cueing (Posner paradigm) Biased competition Visual illusions 0. 08 Dissociative symptoms 0. 06 Sensory attenuation 0. 04 Sensorimotor integration 0. 02 0 Optimal motor control -0. 02 Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses 0. 2 0. 15 Dopamine and affordance 0. 1 Action sequences (reversal learning) 0. 05 Interoceptive inference 0 -0. 05 Communication and hermeneutics 0. 2 0. 4 0. 6 0. 8 1 1. 2 1. 4 time (seconds) Response Local field potentials 0. 25 0. 75 1 1. 25 1. 5 time (seconds) change in precision Phasic dopamine responses 10 20 30 40 50 time (updates) 60 70 80 90

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P 300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions 0. 4 Dissociative symptoms 0. 3 Sensory attenuation 0. 2 Sensorimotor integration 0. 1 0 Optimal motor control -0. 1 Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades 0. 2 Action observation and mirror neuron responses 0. 15 Dopamine and affordance Action sequences (reversal learning) 0. 1 Interoceptive inference 0. 05 Communication and hermeneutics 0 0. 14 Difference waveform (MMN) 0. 12 0. 1 oddball 0. 08 LFP 0. 06 0. 04 standard 0. 02 0 Local field potentials -0. 02 -0. 04 -0. 06 50 MMN 100 150 200 250 Response Time (ms) 0. 25 0. 75 1 1. 25 1. 5 time (seconds) change in precision Phasic dopamine responses 10 20 30 40 50 time (updates) 60 70 80 90 300 350

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P 300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance (transfer of responses) Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P 300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (reversal learning) Interoceptive inference Communication and hermeneutics

What does Bayesian filtering explain? Cross frequency coupling Perceptual categorisation Violation (P 300) responses Oddball (MMN) responses Omission responses Attentional cueing (Posner paradigm) Biased competition Visual illusions Dissociative symptoms Sensory attenuation Sensorimotor integration Optimal motor control Motor gain control Oculomotor control and smooth pursuit Visual searches and saccades Action observation and mirror neuron responses Dopamine and affordance Action sequences (devaluation) Interoceptive inference Communication and hermeneutics

Does the brain use continuous or discrete state space models or both? Control states Hidden states

Thank you And thanks to collaborators: And colleagues: Rick Adams Ryszard Auksztulewicz Andre Bastos Sven Bestmann Harriet Brown Jean Daunizeau Mark Edwards Chris Frith Thomas Fitz. Gerald Xiaosi Gu Stefan Kiebel James Kilner Christoph Mathys Jérémie Mattout Rosalyn Moran Dimitri Ognibene Sasha Ondobaka Will Penny Giovanni Pezzulo Lisa Quattrocki Knight Francesco Rigoli Klaas Stephan Philipp Schwartenbeck Micah Allen Felix Blankenburg Andy Clark Peter Dayan Ray Dolan Allan Hobson Paul Fletcher Pascal Fries Geoffrey Hinton James Hopkins Jakob Hohwy Mateus Joffily Henry Kennedy Simon Mc. Gregor Read Montague Tobias Nolte Anil Seth Mark Solms Paul Verschure And many others