Motor cortex Somatosensory cortex Sensory associative cortex Pars

Motor cortex Somatosensory cortex Sensory associative cortex Pars opercularis Visual associative cortex Broca’s area Visual cortex Primary Auditory cortex Wernicke’s area Language categories and concepts [Adapted from Neural Basis of Thought and Language Jerome Feldman, Spring 2007, feldman@icsi. berkeley. edu

Categories & Prototypes Superordinate Furniture Sofa leather sofa n fabric sofa Desk L-shaped desk Basic-Level Category Reception disk Subordinate Three ways of examining the categories we form: relations between categories (e. g. basic-level category) ¨ internal category structure (e. g. radial category) ¨ instances of category members (e. g. prototypes) ¨

Basic-level -- Criteria n Perception – ¨ overall perceived shape ¨ single mental image ¨ fast identification

Concepts are not categorical

Radial Structure of Mother Genetic mother Stepmother Unwed mother Surrogate mother Biological mother Adoptive mother Central Case Foster mother Birth mother Natural mother The radial structure of this category is defined with respect to the different models

Action Words- an f. MRI study n Somatotopic Representation of Action Words in Human Motor and Premotor Cortex ¨ Olaf Hauk, Ingrid Johnsrude, and Friedemann Pulvermuller* ¨ Medical Research Council, Cognition and Brain Sciences Unit Cambridge, United Kingdom ¨ Neuron, Vol. 41, 1– 20, January 22, 2004, Copyright. 2004 by Cell Press

Traditional theory n Unified meaning center in the left temporal lobe. ¨ Connected to Wernicke’s area ¨ Experiments on highly imageable words/nouns. n Vocalization and grammar associated with frontal lobe ¨ Connected to Broca’s area

Do action words activate the motor cortex n n Given: Cortical representations of the face, arm, and leg are discrete and somatotopically organized in the motor and premotor cortex Hypothesis: Words referring to actions performed with the face, arm, or leg would activate premotor networks. ¨ n neurons processing the word form and those processing the referent action should frequently fire together and thus become more strongly linked, resulting in word-related networks overlapping with motor and premotor cortex in a somatotopic fashion. Experiment: An f. MRI study with word stimuli from different effectors (face, arm, or leg). ROI based on movements (face, arm, leg)

Movement vs. Actions

Neural Evidence for category structure Are there specific regions in the brain to recognize/reason with specific categories? n No, but there are specific circuits distributed over relevant regions of the brain. n

What are schemas? ¨ Regularities in our perceptual, motor and cognitive systems ¨ Structure our experiences and interactions with the world. ¨ May be grounded in a specific cognitive system, but are not situation-specific in their application (can apply to many domains of experience)

Basis of Image schemas n Perceptual systems n Motor routines n Social Cognition n Image Schema properties depend on ¨Neural circuits ¨Interactions with the world

Image schemas n Trajector / Landmark (asymmetric) ¨ ¨ n n bounded region a bounded region has a closed boundary Separation, Contact, Overlap, Inclusion, Surround Orientation Vertical (up/down), Horizontal (left/right, front/back) ¨ Absolute (E, S, W, N) ¨ LM boundary Topological Relations ¨ n The bike is near the house ? The house is near the bike Boundary / Bounded Region ¨ TR

Spatial schemas TR/LM relation n Boundaries, bounded region n Topological relations n Orientational Axes n Proximal/Distal n

TR/LM -- asymmetry The cup is on the table n ? The table is under the cup. n The skateboard is next to the post. n ? The post is next to the skateboard. n

Topological Relations Separation n Contact n Coincidence: n Overlap - Inclusion - - Encircle/surround

Orientation n Vertical axis -- up/down up above below down upright

Relative frame of reference right? ? back front left? ?

Language and Spatial Schemas n n People say that they look up to some people, but look down on others because those we deem worthy of respect are somehow “above” us, and those we deem unworthy are somehow “beneath” us. Much of our language is rich with such spatial talk. Concrete actions such as a push or a lift clearly imply a vertical or horizontal motion, but so too can more abstract concepts. Metaphors: Arguments can go “back and forth, ” and hopes can get “too high. ”

Simulation-based language understanding construction WALKED form selff. phon [wakt] meaning : Walk-Action constraints selfm. time before Context. speech-time selfm. . aspect encapsulated “Harry walked into the cafe. ” Utterance Analysis Process Constructions General Knowledge Semantic Specification Belief State CAFE Simulation

Simulation specification A simulation specification consists of: - schemas evoked by constructions - bindings between schemas

Language and Thought n Language n n Thought cognitive processes n We know thought (our cognitive processes) constrains the way we learn and use language Does language also influence thought? Benjamin Whorf argues yes Psycholinguistics experiments have shown that linguistics categories influence thinking even in non-linguistics task

Natural Theory of Language (NTL) • Basic concepts and words derive their meaning from embodied experience. • Abstract and theoretical concepts derive their meaning from metaphorical maps to more basic embodied concepts. • Structured Connectionist Models can capture both of these processes nicely. • Grammar extends this by Constructions: pairings of form with embodied meaning.

Simulation-based language understanding “Harry walked to the cafe. ” Constructions General Knowledge Belief State Utterance Analysis Process Schema walk Trajector Harry Cafe Goal cafe Simulation Specification Simulation

The ICSI/Berkeley Neural Theory of Language Project

Background: Primate Motor Control n Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod, Rizzolatti) Should model coordinated, distributed, parameterized control programs required for motor action and perception. ¨ Should be an active structure. ¨ Should be able to model concurrent actions and interrupts. ¨ n Model The NTL project has developed a computational model based on that satisfies these requirements (x- schemas). ¨ Details, papers, etc. can be obtained on the web at http: //www. icsi. berkeley. edu/NTL ¨

Active representations n n n Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions Many inferences about actions derive from what we know about executing them Generative model: action, planning, recognition, language. walker at goal energy walker=Harry goal=home Walking: bound to a specific walker with a direction or goal consumes resources (e. g. , energy) may have termination condition (e. g. , walker at goal) ongoing, iterative action

Language Development in Children n n n n 0 -3 mo: prefers sounds in native language 3 -6 mo: imitation of vowel sounds only 6 -8 mo: babbling in consonant-vowel segments 8 -10 mo: word comprehension, starts to lose sensitivity to consonants outside native language 12 -13 mo: word production (naming) 16 -20 mo: word combinations, relational words (verbs, adj. ) 24 -36 mo: grammaticization, inflectional morphology 3 years – adulthood: vocab. growth, sentence-level grammar for discourse purposes

Learning Spatial Relation Words Terry Regier A model of children learning spatial relations. Assumes child hears one word label of scene. Program learns well enough to label novel scenes correctly. Extended to simple motion scenarios, like INTO. System works across languages. Mechanisms are neurally plausible.

Learning Verb Meanings David Bailey A model of children learning their first verbs. Assumes parent labels child’s actions. Child associates action with word Program learns well enough to: 1) Label novel actions correctly 2) Obey commands using new words (simulation) System works across languages Mechanisms are neurally plausible.

Motor Control (X-schema) for SLIDE

Parameters for the SLIDE Xschema

Training Results David Bailey English n n n 165 Training Examples, 18 verbs Learns optimal number of word senses (21) 32 Test examples : 78% recognition, 81% action All mistakes were close lift ~ yank, etc. Learned some particle CXN, e. g. , pull up Farsi n With identical settings, learned senses not in English

Embodied Construction Grammar (Bergen, Chang & Paskin 2000) n Assumptions from Construction Grammar ¨ Constructions are form-meaning pairs (Lakoff 1987, Goldberg 1995) ¨ Constructions vary in degree of specificity and level of description (morphological, lexical, phrasal, clausal) n n Constructions evoke and bind semantic schemas Additional influences ¨ Cognitive Grammar (Langacker 1987) ¨ Frame Semantics (Fillmore 1982) ¨ Structured Connectionism (Feldman 1987)

Conclusion n Language acquisition and use is a hallmark of being human ¨ Language seems to rely on n fine-grained aspects of embodied (sensorymotor and social cognition) primitives and n brain-like computation (massively parallel, distributed, spreading activation, temporal binding). ¨ Understanding requires imaginative simulation! n Sensory-Motor imagination and simulation is crucial in interpretation!