A Hierarchical Neural Model for the Detection of

A Hierarchical Neural Model for the Detection of Motion Patterns in Optical Flow Fields Overview 1. The Structure of the Visual Cortex 2. Using Selective Tuning to Model Visual Attention 3. The Motion Hierarchy Model 4. Simulation Results 5. Conclusions

“Data Flow Diagram” of Visual Areas in Macaque Brain Blue: motion perception pathway Green: object recognition pathway

Receptive Fields in Hierarchical Neural Networks neuron A receptive field of A

Receptive Fields in Hierarchical Neural Networks neuron A in top layer receptive field of A in input layer

Problems with Information Routing in Hierarchical Networks contextual interference poor localization crosstalk

The Selective Tuning Concept (Tsotsos, 1988) unit of interest at top processing pyramid pass pathways: hierarchical restriction of input space input inhibited pathways

Hierarchical Winner-Take-All WTA achieved through local gating networks top-down, coarse-to-fine WTA hierarchy for selection and localization unselected connections are inhibited

Selection Circuits layer l+1 unit and connection in the interpretive network I l+1, x unit and connection in the gating network unit and connection in the top-down bias network layer l B l+1, k I l, k bl, k gl, k U l+1, k }M l, k G l, k, j layer l-1 I l-1, j

Two-Phase WTA for Region Selection Phase 1: • distance-invariant • minimum difference in activation necessary for inhibition • problem: possible split-up of attended regions Phase 2: • mutual inhibition grows with the distance between units • only one coherent region is selected for attention

3 D Visualization of the Selective Tuning Network Red: WTA phase 1 active Blue: inhibition Green: WTA phase 2 active Yellow: WTA winner

The Motion Perception Pathway MST feedforward feedback MT feedforward feedback V 1 feedforward feedback input

What do We Know about Area V 1? • cells have small receptive fields • each cell has a preferred direction of motion activation preferred direction of motion • there are three types of motion speed selectivity activation low-speed cells medium-speed cells high-speed cells speed of motion

What do We Know about Area MT? • cells have larger receptive fields than in V 1 • like in V 1, each cell has a preferred combination of the direction and speed of motion • MT cells also have a preferred orientation of the speed gradient activation preferred orientation of speed gradient without speed gradient with speed gradient orientation of speed gradient

What do We Know about Area MST? • cells respond to motion patterns such as – translation (objects shifting positions) – rotation (clockwise and counterclockwise) – expansion (approaching objects) – contraction (receding objects) – spiral motion (combinations of rotation and expansion/contraction) • the response of a cell is almost independent on the position of the motion pattern in the visual field

The Motion Hierarchy Model: V 1 • V 1 receives optical flow patterns as input counterclockwise contraction expansion rotation counterclockwise contraction expansion

The Motion Hierarchy Model: V 1 • V 1 is simulated as 60 60 hypercolumns • each column contains 36 cells: one for each combination of direction (12) and speed tuning (3) • direction and speed selectivity are modeled with Gaussian functions based on physiological data • the activation of a V 1 cell is the product of its activation by direction and its activation by speed • example: cells tuned towards upward motion: high-speed cells medium-speed cells input pattern: counter-clockwise rotation low-speed cells

The Motion Hierarchy Model: MT • MT is simulated as 30 30 hypercolumns • each column contains 432 cells: one for each combination of direction (12) speed (3), and speed gradient tuning (12) • problem: how can gradient tuning be realized from activation patterns in V 1? – solution: detect gradient differences across the three types of speed selective cells – this solution leads to a simple network structure and remarkably good noise reduction • the activation of an MT cell is the product of its activation by direction, speed, and gradient

The Motion Hierarchy Model: MT • structure of input connections to MT cells: MT V 1 • if the input is a counterclockwise rotation, these MT cells respond to – medium speed – leftward motion – upward speed gradient

The Motion Hierarchy Model: MST • how can MST cells detect motion patterns such as rotation, expansion, and contraction based on the activation of MT cells? movement counterclockwise speed gradient contraction expansion • idea: the presence of these motion patterns is indicated by a consistent angle between the local movement and speed gradient

The Motion Hierarchy Model: MST direction of movement orientation of speed gradient

The Motion Hierarchy Model: MST • MST cells integrate the activation of MT cells that respond to a particular angle between motion and speed gradient • this integration is performed across a large part of the visual field and across all 12 directions • therefore, MST can detect 12 different motion patterns • we simulate 5 5 MST hypercolumns, each containing 36 neurons (tuned for 12 different motion patterns, 3 different speeds)

The Motion Hierarchy Model: MST “wiring” of MST cells tuned for clockwise rotation MT speed gradient tuning MST cells MT motion direction tuning

Simulation: clockwise rotation

Simulation: counterclockwise rotation

Simulation: receding object

Attention in the Motion Hierarchy What happens if there are multiple motion patterns in the visual input? Visual attention can be used to • determine the type and location of the most salient motion pattern, • focus on it by eliminating all interfering information, • sequentially inspect all objects in the visual field.

Attention in the Motion Hierarchy Iterative application of the attentional mechanism:

Conclusions and Outlook • the motion hierarchy model provides a plausible explanation for cell properties in areas V 1, MT, and MST • its use of distinct speed tuning functions in V 1 and speed gradient selectivity in MT leads to a relatively simple network structure combined with robust and precise detection of motion patterns • visual attention is employed to segregate and sequentially inspect multiple motion patterns

Conclusions and Outlook • the model is well-suited for mobile robots to estimate parameters of self-motion • the area MST in the simulated hierarchy is very sensitive to any translational or rotational self-motion • in biological vision, MST is massively connected to the vestibular system • in mobile robots, the simulated area MST could interact with position and orientation sensors to stabilize self-motion estimation