Cog RIC Workshop Adaptive Working Memory From Computational

Cog. RIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer Science Assistant Professor of Psychology Vanderbilt University david. noelle@vanderbilt. edu August 17, 2006

Cog. RIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer Science Assistant Professor of Cognitive Science University of California, Merced david. noelle@vanderbilt. edu August 17, 2006

Cog. RIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer Science Assistant Professor of Cognitive Science University of California, Merced david. noelle@vanderbilt. edu August 17, 2006

Adaptive Working Memory Project Funded by NSF ITR program (EIA-0325641) www. cecs. missouri. edu/~skubic/WM/ David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Adaptive Working Memory Project Funded by NSF ITR program (EIA-0325641) www. cecs. missouri. edu/~skubic/WM/ Kaz Kawamura Mitch Wilkes Marge Skubic Jim Keller David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Working Memory David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Working Memory Working memory systems are those that actively maintain transient information that is critical for successful decision-making in the current context. A working memory system can be viewed as a relatively small cache of task relevant information that is strategically positioned to efficiently influence behavior. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Working Memory In The Brain ● ● A number of brain regions are implicated as important components of the human working memory system. One important region is dorsolateral portions of prefrontal cortex. Working memory is exhibited in delay period activity. Cells have been found which encode for locations, visual features, and association rules. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Modeling Collaborators David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Active Maintenance ● How are high neural firing rates sustained over a delay? ● Mutual excitation of neurons. ● ● Dense recurrent connections in prefrontal cortex. Stripe sets. Attractor network computational models. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Controlling PFC Updating How does PFC know when to actively maintain its current working memory contents? How does it know when to abandon a given working memory “chunk” in favor of a new one? The dynamics of recurrent attractor networks are insufficient to meet the simultaneous constraints of (1) active maintenance in the face of distraction and (2) rapid updating when needed. A dynamic gating mechanism is needed. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Mesolimbic Dopamine (DA) System David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Dopamine (DA) Cells (Schultz, Apicella, and Ljungberg, 1993) David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Temporal Difference Learning ● ● DA neurons seem to encode for change in expected reward. This is equivalent to the key variable, called temporal difference error, in a powerful reinforcement learning algorithm called temporal difference (TD) learning. The brain may learn to produce rewarding motor sequences using a neural implementation of TD learning (Montague, Dayan, and Sejnowski, 1996). There are extensive DA projections to PFC. If TD learning is used to learn when to produce a given overt action, perhaps it can be used to learn when to produce a covert action – like updating working memory (Braver and Cohen, 2000). David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Computational Cognitive Neuroscience Models ● ● ● Healthy performance on frontal tasks. Prolonged frontal developmental period. Monkey lesion data. Human frontal damage patient performance. Autistic peformance. (Rougier, Noelle, Braver, Cohen, and O'Reilly, 2005) David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Robotic Working Memory Could robot control systems benefit from the inclusion of a working memory system? ● ● The highly limited capacity of working memory, along with its tight coupling with deliberation mechanisms, might alleviate the need for costly memory searches. Information needed to fluently perform the current task is temporarily kept “handy” in the working memory store. Can computational neuroscience models of the working memory mechanisms of the human brain shed light on the design of a robotic working memory system? David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Potential Uses ● ● Focus attention on the most relevant features of the current task. Guide perceptual processes by limited the perceptual search space. Provide a focused short-term memory to prevent the robot from being confused by occlusions. Provide robust operation in the presence of distracting irrelevant events. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Adaptive Working Memory How does the working memory system know when a given chunk of information should be actively maintained in working memory? ● ● Hand Coding – For relatively routine and well understood tasks, designers may hand code procedures for the identification of useful chunks. Learning – If the robot is expected to flexibly respond in novel task situations, or even acquire new tasks, it would be beneficial to have a means to learn when to store a particular chunk in working memory. The central focus of this project is on assessing the utility of adaptive working memory mechanisms for robot control. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

The Working Memory Toolkit ● ● ● Memory traces, or chunks, are pointers to arbitrary C++ data structures. The adaptive working memory toolkit (WMtk) requires the user to specify: the capacity of the working memory a function which extracts features from chunks a function which provides relevant features of the current system state a function which provides instantaneous external reward information The toolkit provides a function for examining the contents of working memory, returning chunk pointers. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

On Each Time Step. . . ● ● ● The robot control system making use of the WMtk suggests candidate chunks for retention by the working memory. A component of the TD learning algorithm, called the adaptive critic, is used to estimate the expected future reward of retaining various combinations of chunks. The collection of chunks with the highest expected future reward value are remembered (with high probability). The amount of instantaneous external reward received on this time step, along with the estimates of the adaptive critic on this time step and on the previous time step, are used to compute the TD error – the change in expected future reward. This error signal is used to train the adaptive critic. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Delayed Saccade Task David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Robotic Delayed Saccade Task David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Task Structure ● ● ● Three kinds of “goal” working memory chunks: Stay fixated on the object that you are looking at. Look at the last location of the crosshair. Look at the last location of the target. The robot control system obeys any goal chunks in working memory (resolving “look at” conflicts at random). If no chunks are being actively maintained, the system looks at a randomly selected object or, when there are no displayed objects, at a random location on the screen. Note that remembering all chunks will often lead to failure. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Results The robot successfully learns the task. . . after about 4000 trials. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Revisiting The Neuroscience ● ● Dopamine cells sometimes fire in a way that does not reflect a change in expected future reward. Specifically, they often fire in novel situations. If the dopamine signal is seen as TD error, this suggests that brain treats novel situations as if they were more predictive of reward than is warranted by experience. This has been implemented in the WMtk through the incorporation of an optimistic critic – the TD algorithm is initialized to predict high future reward for novel combinations of chunks. With this modification. . . David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Optimistic Critic Results The robot successfully learns the task. . . after about 300 trials. An improvement by more than a factor of ten! David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Other Preliminary Successes ● ● Mitch Wilkes and his students have used the WMtk to allow a physical mobile robot to. . . . learn which percepts to approach in order to produce the largest amount of forward motion down a hallway. . learn which percepts reliably identify a specific target location, where reward is received. Marge Skubic and her students have used the WMtk to allow a simulated robot to learn which motor program to retain as a goal chunk, given the current sensory state, so as to solve a water maze problem. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Summary ● ● ● Computational cognitive neuroscience models of the interactions between the prefrontal cortex and the midbrain dopamine system have been successful at accounting for a variety of working memory phenomena. The basic structure of these models, involving the use of a reinforcement learning algorithm to learn, from experience, what should be retained in working memory and what can be safely forgotten, has been abstracted into an open source software library called the Working Memory Toolkit. By attending to nuances of biology, the adaptive learning capabilities of the toolkit have been greatly improved. David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Questions? David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

The End David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

Extra Slides David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

PFC Stripes & Thalamic Loops David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle

A More Complex Network David C. Noelle, Ph. D. david. noelle@vanderbilt. edu people. vanderbilt. edu/~david. noelle