Dealing with Implicit Negatives n Explicit positive for

Dealing with Implicit Negatives n Explicit positive for above Implicit negatives for below, left, right, etc n in Regier: n E = ½ ∑i, p (( ti, p – oi, p) * βi, p )2, where i is the node, p is the pattern, βi, p = 1 if explicit positive, βi, p < 1 if implicit negative

Learning System dynamic relations (e. g. into) structured connectionist network (based on visual system)

Topological Relations Separation n Contact n Coincidence: n Overlap - Inclusion - - Encircle/surround

Issue #2: Shift Invariance n n n Backprop cannot handle shift invariance (it cannot generalize from 0011, 0110 to 1100) But the cup is on the table whether you see it right in the center or from the corner of your eyes (i. e. in different areas of the retina map) What structure can we utilize to make the input shift-invariant?

Limitations n Scale n Uniqueness/Plausibility n Grammar n Abstract Concepts n Inference n Representation

Demo of the Regier System on the English above

Language and Thought n Language n n n Thought cognitive processes We know thought (our cognitive processes) constrains the way we learn and use language Does language also influence thought? Benjamin Whorf argues yes Psycholinguistics experiments have shown that linguistics categories influence thinking even in non-linguistics task

5 levels of Neural Theory of Language Spatial Relation Motor Control Metaphor Grammar Cognition and Language abstraction Computation Structured Connectionism Neural Net SHRUTI Computational Neurobiology Triangle Nodes Biology Neural Development Quiz Midterm Finals

Language, Learning and Neural Modeling www. icsi. berkeley. edu/AI n Scientific Goal Understand how people learn and use language n Practical Goal Deploy systems that analyze and produce language n Approach Build models that perform cognitive tasks, respecting all experimental and experiential constraints Embodied linguistic theories with advanced biologically-based computational methods

Constrained Best Fit in Nature inanimate physics chemistry biology vision language animate lowest energy state molecular minima fitness, MEU Neuroeconomics threats, friends errors, NTL

Simulation-based language understanding “Harry walked to the cafe. ” Constructions General Knowledge Belief State Utterance Analysis Process Schema walk Trajector Harry Cafe Goal cafe Simulation Specification Simulation

Simulation Semantics n n n BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE ¨ Evidence for common mechanisms for recognition and action (mirror neurons) in the F 5 area (Rizzolatti et al (1996), Gallese 96, Boccino 2002) and from motor imagery (Jeannerod 1996) IMPLEMENTATION: ¨ x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x -schemas. State is completely distributed (a graph marking) over the network. RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

Psycholinguistic evidence n Embodied language impairs action/perception ¨ Sentences with visual components to their meaning can interfere with performance of visual tasks (Richardson et al. 2003) ¨ Sentences describing motion can interfere with performance of incompatible motor actions (Glenberg and Kashak 2002) ¨ Sentences describing incompatible visual imagery impedes decision task (Zwaan et al. 2002) n Simulation effects from fictive motion sentences ¨ Fictive motion sentences describing paths that require longer time, span a greater distance, or involve more obstacles impede decision task (Matlock 2000, Matlock et al. 2003)

Neural evidence: Mirror neurons n Gallese et al. (1996) found “mirror” neurons in the monkey motor cortex, activated when ¨ an action was carried out ¨ the same action (or a similar one) was seen. n Mirror neuron circuits found in humans (Porro et al. 1996) n Mirror neurons activated when someone: ¨ imagines an action being carried out (Wheeler et al. 2000) ¨ watches an action being carried out (with or without object) (Buccino et al. 2000)

Area F 5 c Convexity region of F 5: Mirror neurons

F 5 Mirror Neurons Gallese and Goldman, TICS 1998

Category Loosening in Mirror Neurons (~60%) Observed: A is Precision Grip B is Whole Hand Prehension Action: C: precision grip D: Whole Hand Prehension (Gallese et al. Brain 1996)

PF Mirror Neurons 1. 2. 3. Neuron responds to execution (grasping) but to grasping and releasing in observation. Mirror neurons in parietal cortex. Difference in left hand right hand. (Gallese et al. 2002)

A (Full vision) B (Hidden) C (Mimicking) D (Hidden. Mimicking) Umiltà et al. Neuron 2001

F 5 Audio-Visual Mirror Neurons Kohler et al. Science (2002)

Somatotopy of Action Observation Foot Action Hand Action Mouth Action Buccino et al. Eur J Neurosci 2001

The Mirror System in Humans BA 6

The ICSI/Berkeley Neural Theory of Language Project ECG Learning early constructions (Chang, Mok)

Computing other relations The 2/3 node is a useful function that activates its outputs (3) if any (2) of its 3 inputs are active n Such a node is also called a triangle node and will be useful for lots of representations. n

Triangle nodes and Mc. Cullough. Pitts Neurons? A B C

Representing concepts using triangle nodes: when two of the neurons fire, the third also fires

“They all rose” triangle nodes: when two of the neurons fire, the third also fires model of spreading activation

Basic Ideas behind the model n n n Parallel activation streams. Top down and bottom up activation combine to determine the best matching structure. Triangle nodes bind features of objects to values Mutual inhibition and competition between structures Mental connections are active neural connections

Can we formalize/model these intuitions What is a neurally plausible computational model of spreading activation that captures these features. n What does semantics mean in neurally embodied terms n ¨ What are the neural substrates of concepts that underlie verbs, nouns, spatial predicates?

Spreading activation and feature structures n n n Parallel activation streams. Top down and bottom up activation combine to determine the best matching structure. Triangle nodes bind features of objects to values Mutual inhibition and competition between structures Mental connections are active neural connections

Feature Structures in Four Domains Barrett Ham Container Push dept~CS Color ~pink Inside ~region Schema ~slide sid~001 Taste ~salty Outside ~region Posture ~palm Bdy. ~curve Dir. ~ away emp~GSI Chang Pea Purchase Stroll dept~Ling Color ~green Buyer ~person Schema ~walk sid~002 Taste ~sweet Seller ~person Speed ~slow Cost ~money Dir. ~ ANY emp~Gra Goods ~ thing