The ICSIBerkeley Neural Theory of Language Project ECG

The ICSI/Berkeley Neural Theory of Language Project ECG Learning early constructions (Chang, Mok)

Connectionist Model of Word Recognition (Rumelhart and Mc. Clelland)

Constraints on Connectionist Models 100 Step Rule Human reaction times ~ 100 milliseconds Neural signaling time ~ 1 millisecond Simple messages between neurons Long connections are rare No new connections during learning Developmentally plausible

Can we formalize/model these intuitions What is a neurally plausible computational model of spreading activation that captures these features. n What does semantics mean in neurally embodied terms n ¨ What are the neural substrates of concepts that underlie verbs, nouns, spatial predicates?

Abstract Neuron output y net > 0 {10 ifotherwise w 0 I 0 = 1 w 1 i 1 w 2 i 2 wn. . . input i in

Computing with Abstract Neurons n Mc. Collough-Pitts Neurons were initially used to model ¨ pattern classification n size = small AND shape = round AND color = green AND location = on_tree => unripe ¨ linking classified patterns to behavior n size = large OR motion = approaching => move_away n size = small AND direction = above => move_above n Mc. Collough-Pitts Neurons can compute logical functions. ¨ AND, NOT, OR

Distributed vs Localist Rep’n John 1 1 0 0 John 1 0 0 0 Paul 0 1 1 0 Paul 0 1 0 0 George 0 0 1 1 George 0 0 1 0 Ringo 1 0 0 1 Ringo 0 0 0 1 What are the drawbacks of each representation?

Distributed vs Localist Rep’n n n John 1 1 0 0 John 1 0 0 0 Paul 0 1 1 0 Paul 0 1 0 0 George 0 0 1 1 George 0 0 1 0 Ringo 1 0 0 1 Ringo 0 0 0 1 What happens if you want to represent a group? How many persons can you represent with n bits? 2^n n n What happens if one neuron dies? How many persons can you represent with n bits? n

Sparse Distributed Representation

Visual System n n 1000 x 1000 visual map For each location, encode: ¨orientation ¨direction … of motion ¨speed ¨size ¨color ¨depth n Blows up combinatorically! …

Coarse Coding info you can encode with one fine resolution unit = info you can encode with a few coarse resolution units Now as long as we need fewer coarse units total, we’re good

Coarse-Fine Coding Coarse in F 2, Fine in F 1 Feature 1 e. g. Orientation Y-Orientation Y G X-Orientation G X Y-Dir X-Dir Coarse in F 1, Fine in F 2 but we can run into ghost “images” Feature 2 e. g. Direction of Motion

Connectionist Models in Cognitive Science Structured PDP Hybrid Neural Conceptual Existence Data Fitting

Computing other relations The 2/3 node is a useful function that activates its outputs (3) if any (2) of its 3 inputs are active n Such a node is also called a triangle node and will be useful for lots of representations. n

Triangle nodes and Mc. Cullough. Pitts Neurons? A B C

“They all rose” triangle nodes: when two of the neurons fire, the third also fires model of spreading activation

Spreading activation and feature structures n n n Parallel activation streams. Top down and bottom up activation combine to determine the best matching structure. Triangle nodes bind features of objects to values Mutual inhibition and competition between structures Mental connections are active neural connections

Representing concepts using triangle nodes

Feature Structures in Four Domains Barrett Ham Container Push dept~CS Color ~pink Inside ~region Schema ~slide sid~001 Taste ~salty Outside ~region Posture ~palm emp~GSI Bdy. ~curve Dir. ~ away Schneider Pea Purchase Stroll dept~Ling Color ~green Buyer ~person Schema ~walk sid~002 Taste ~sweet Seller ~person Speed ~slow Cost ~money Dir. ~ ANY emp~Gra Goods ~ thing

Categories and conceptsintroduction CS 182/Ling 109/Cog. Sci 110 Spring 2008

Lecture Outline n Categories ¨ Basic Level ¨ Prototype Effects ¨ Neural Evidence for Category Structure n n Aspects of a Neural Theory of concepts Image Schemas ¨ Description and types ¨ Behavioral Experiment on Image Schemas n Event Structure and Motor Schemas

The WCS Color Chips n Basic color terms: ¨ Single word (not blue-green) ¨ Frequently used (not mauve) ¨ Refers primarily to colors (not lime) ¨ Applies to any object (not blonde)

Concepts n What Concepts Are: Basic Constraints ¨ Concepts are the elements of reason, and ¨ constitute the meanings of words and linguistic expressions.

Concepts Are: • Universal: they characterize all particular instances; e. g. , the concept of grasping is the same no matter who the agent is or what the patient is or how it is done. • Stable. • Internally structured. • Compositional. • Inferential. They interact to give rise to inferences. • Relational. They may be related by hyponymy, antonymy, etc. • Meaningful. • Not tied to the specific word forms used to express them.

Concepts: Traditional Theory n The Traditional Theory ¨ Reason and language are what distinguish human beings from other animals. ¨ Concepts therefore use only human-specific brain mechanisms. ¨ Reason is separate from perception and action, and does not make direct use of the sensory-motor system. ¨ Concepts must be “disembodied” in this sense.

The neural theory Human concepts are embodied. Many concepts make direct use of sensorymotor, emotional, and social cognition capacities of our body-brain system. n Many of these capacities are also present in non-human primates. n Continuity Principle of Am. Pragmatists

Classical vs prototype model of categorization n Classical model ¨ Category membership determined on basis of essential features ¨ Categories have clear boundaries ¨ Category features are binary n Prototype model ¨ Features that frequently co-occur lead to establishment of category ¨ Categories are formed through experience with exemplars

Prototype theory 1. 2. 3. 4. 5. 6. Certain members of a category are prototypical – or instantiate the prototype Categories form around prototypes; new members added on basis of resemblance to prototype No requirement that a property or set of properties be shared by all members Features/attributes generally gradable Category membership a matter of degree Categories do not have clear boundaries

Prototype theory 1. Certain members of a category are prototypical – or instantiate the prototype Category members are not all equal a robin is a prototypical bird, but we may not want to say it is the prototype, rather it instantiates (manifests) the prototype or ideal -it exhibits many of the features that the abstract prototype does “It is conceivable that the prototype for dog will be unspecified for sex; yet each exemplar is necessarily either male or female. ” (Taylor)

Prototype theory 3. No requirement that a property or set of properties be shared by all members -- no criterial attributes ¨ ¨ Category where a set of necessary and sufficient attributes can be found is the exception rather than the rule Labov household dishes experiment n n Necessary that cups be containers, not sufficient since many things are containers Cups can’t be defined by material used, shape, presence of handles or function

Prototype theory ¨ Wittgenstein’s examination of game n Generally necessary that all games be amusing, not sufficient since many things are amusing n Board games, ball games, card games, etc. have different objectives, call on different skills and motor routines - categories normally not definable in terms of necessary and sufficient features

Prototype theory n What about mathematical categories like odd or even numbers? Aren’t these sharply defined? ¨ (Armstrong et al. ) Subjects asked to assign numbers a degree of membership to the categories odd number or even number 3 had a high degree of membership, 447 and 91 had a lower degree (all were rated at least ‘moderately good’)

Categories - who decides? Embodied theory of meaning- categories are not pre-formed and waiting for us to behold them. Our need for categories drives what categories we will have n Basic level categories - not all categories have equal status. The basic level category has demonstrably greater psychological significance. n

Basic-level categories

furniture Superordinate chair desk chair easy chair rocking chair lamp desk lamp floor lamp table dining room table coffee table Basic Subordinate

Categories & Prototypes: Overview Superordinate Furniture Sofa leather sofa n fabric sofa Desk L-shaped desk Basic-Level Category Reception disk Subordinate Three ways of examining the categories we form: relations between categories (e. g. basic-level category) ¨ internal category structure (e. g. radial category) ¨ instances of category members (e. g. prototypes) ¨

Basic-level -- Criteria n Perception – ¨ overall perceived shape ¨ single mental image ¨ fast identification

Basic-level -- Criteria Perception n Function – motor program for interaction n

Basic-level -- Criteria Perception n Function n Words – n shortest ¨ first learned by children ¨ first to enter lexicon ¨

Basic-level -- Criteria Perception n Function n Communication n Knowledge organization – n ¨ most attributes are stored at this level

Basic-Level Category What constitutes a basic-level category? n Perception: n shortest ¨ most commonly used ¨ contextually neutral ¨ first to be learned by children ¨ first to enter the lexicon similar overall perceived shape ¨ single mental image ¨ (gestalt perception) ¨ fast identification ¨ ¨ n Function: ¨ general motor program Communication: n Knowledge Organization: ¨ most attributes of category members stored at this level

Other Basic-level categories Objects n Colors n Motor-routines n

Concepts are not categorical

Mother n The birth model The person who gives birth is the mother n The genetic model The female who contributes the genetic material is the mother n The nurturance model The female adult who nurtures and raises a child is the mother of the child n The marital model The wife of the father is the mother n The genealogical model The closest female ancestor is the mother (WFDT Ch. 4, p. 74, p. 83)

Radial Structure of Mother Genetic mother Stepmother Unwed mother Surrogate mother Biological mother Adoptive mother Central Case Foster mother Birth mother Natural mother The radial structure of this category is defined with respect to the different models

Marriage n What is a marriage? n What are the frames (or models) that go into defining a marriage? n What are prototypes of marriage? n What metaphors do we use to talk about marriages? n Why is this a contested concept right now?

Concepts and radial categories Concepts can get to be the "prototype" of their category in various ways. n Central subcategory (others relate to this) n n n Essential (meets a folk definition: birds have feathers, beaks, lay eggs) n n Move involves change of location. Typical case (most are like this: "sparrow") n n Amble and swagger relate to WALK Shove relates to PUSH Going to a conference involves air travel. Ideal/anti-ideal case (positive social standard: "parent"); anti-ideal case (negative social standard: "terrorist") Stereotype (set of attributes assumed in a culture: "Arab") Salient exemplar (individual chosen as example)

Category Structure n Classical Category: ¨ n necessary and sufficient conditions Radial Category: a central member branching out to less-central and non-central cases ¨ degrees of membership, with extendable boundary ¨ n Family Resemblance: every family member looks like some other family member(s) ¨ there is no one property common across all members (e. g. polysemy) ¨ n n Prototype-Based Category Essentially-Contested Category (Gallie, 1956) (e. g. democracy) n Ad-hoc Category (e. g. things you can fit inside a shopping bag)

Prototype n Cognitive reference point ¨ n n standards of comparison e. g. ideal vacation ¨ can be abstract ¨ may be neither typical nor stereotypical ¨ Social stereotypes snap judgments ¨ defines cultural expectations ¨ challengeable ¨ n n default expectation ¨ often used unconsciously in reasoning Paragons / Anti-paragons ¨ Typical case prototypes ¨ Ideal case / Nightmare case n Salient examples ¨ n an individual member that exhibits the ideal e. g. 9/11 – terrorism act Generators central member + rules ¨ e. g. natural number = singledigit numbers + arithmetic ¨

Neural Evidence for category structure n Are there specific regions in the brain to recognize/reason with specific categories?

Category Naming and Deficits People with brain injury have selective deficits in their knowledge of categories. n Some patients are unable to identify or name man made objects and others may not be able to identify or name natural kinds (like animals) n

A PET Study on categories (Nature 1996)

Study n 16 adults (8 M, 8 F) participated in a PET (positron emission tomography) study. Involves injecting subject with a positron emitting radioactive substance (dye) ¨ Regions with more metabolic activity will absorb more of the substance and thus emit more positrons ¨ Positron-electron collisions yield gamma rays, which are detected ¨ n Increased r. CBF (regional changes in cerebral blood flow) was measured ¨ When subjects viewed line drawings of animals and tools.

The experiment n n n Subjects looked at pictures of animals and tools and named them silently. They also looked at noise patterns (baseline 1) And novel nonsense objects (baseline 2) Each stimulus was presented for 180 ms followed by a fixation cross of 1820 ms. Drawings were controlled for name frequency and category typicality

medial lateral

Left middle temporal gyrus ACC Premotor

Calcarine Sulcus

Conclusions n n Both animal and tool naming activate the ventral temporal lobe region. Tools differentially activate the ACC, pre-motor and left middle temporal region (known to be related to processing action words). Naming animals differentially activated left medial occipital lobe (early visual processing) The object categories appear to be in a distributed circuit that involves activating different salient aspects of the category.