Bailey Model Showed how simple hand action verbs

Bailey Model • Showed how simple hand action verbs may be acquired based on motor control schemas and parameterization. • Used Model Merging which allowed for – One-shot learning (Maps to recruitment learning) • Could label and perform actions (given a command, world state pair) • Uses parameters over motor-control schemas as inductive bias • Limitations – Inference • Connections between events – Abstract uses • Event Structure • Frames • Metaphor

Event Structure-1 Srini Narayanan CS 182/Cog. Sci 110/Ling 109 Spring 2006 snarayan@icsi. berkeley. edu

Active representations • Many inferences about actions derive from what we know about executing them • Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions walker at goal energy walker=Harry goal=home Walking: bound to a specific walker with a direction or goal consumes resources (e. g. , energy) may have termination condition (e. g. , walker at goal) ongoing, iterative action

X-Schema Extensions to Petri Nets • Parameterization – x-schemas take parameter values (speed, force) • Walk(speed = slow, dest = store 1) • Dynamic Binding – X-schemas allow run-time binding to different objects/entities • Grasp(cup 1), push(cart 1) • Hierarchical control and durative transitions – Walk is composed of steps which are composed of stance and swing phases • Stochasticity and Inhibition – Uncertainties in world evolution and in action selection

Event Structure in Language • Commonplace discourse fragments/blurbs – Low inflation is starting to pull France out of recession. – E 3 continue to push Iran to uphold IAEA obligations. – US Economy on the verge of falling back into recession after moving forward on an anemic recovery. – Indian Government stumbling in implementing Liberalization plan. – Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice. – The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.

Event Structure in Language • Fine-grained • Rich Notion of Contingency Relationships. – Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals • Event Structure Metaphor: – Phenomena: Abstract Actions are conceptualized in Motion and Manipulation terms. – Schematic Inferences are preserved.

Aspect • Aspect is the name given to the ways languages describe the structure of events using a variety of lexical and grammatical devices. – Viewpoints • is walking, walk – Phases of events • Starting to walk, walking, finish walking – Inherent Aspect • run vs cough vs. rub – Composition with • Temporal modifiers, tense. . • Noun Phrases (count vs. mass) etc. .

Grammatical Aspect Languages have grammatical constructions that indicate the type of situation described. • • • Progressive: She was running home. Perfect: I’ve had a wonderful evening. Inceptive: She started knitting. Prospective: She’s about to leave. Resumptive: Peace talks resume. Iterative: They ran twice around the track.

Phases, Viewpoints, and Aspects • • • John is walking to the store. John is about to walk to the store. John walked to the store. John started walking to the store. John is starting to walk to the store. John has walked to the store. John has started to walk to the store. John is about to start walking to the store. John resumed walking to the store. John has been walking to the store. John has finished walking to the store. John almost walked to the store.

A Walk X-schema

A Climb X-schema

Common Patterns START FINISH Posture = Up Energy Available Ready Dest = top(obj) Loop BEGIN Execute (subschema) END At Dest Done Posture = Up Ground ok Ready Loop BEGIN Execute(subschema) END At Dest Done

Pre-motor Versus Motor Cortex Whenever we perform a complex motor movement, such as picking up a glass and taking a drink, at least two distinct parts of the brain are activated: The motor cortex, where there are neural ensembles that control “motor synergies” — relatively simple actions like opening or closing the hand, flexing or extending the elbow, turning the wrist, and so on. Complex motor schemas, however, are carried out by neural circuitry in the pre-motor cortex, circuitry connected via neural bindings to the appropriate synergies in the motor cortex. In picking up a glass and taking a drink, both pre-motor cortex and motor cortex are activated, as are binding connections between them.

The Controller X-Schema In modeling complex premotor action schemas, we make the following hypothesis All complex premotor schemas are compositions of a single type of structure. The same neural computational structure, when disengaged from the motor cortex, can characterize aspect (that is, event structure) in the world’s languages. When dynamically active, this structure can compute the logic of aspect. We call this structure the “Controller X-schema. ”

The Structure of the Controller X-Schema • Initial State • Starting Phase Transition • Precentral State • Central Phase Transition (either instantaneous, prolonged, or ongoing) • Postcentral State* • Ending Phase Transition • Final State Postcentral Options: *A check to see if a goal state has been achieved *An option to stop/resume *An option to iterate or continue the main process -Narayanan, 1997

A Schema Controller iterate Ready Start Cancel Process Finish interrupt resume Done Suspend • An active controller that sends signals to the embedded schema and transitions based on signals from the embedded schema. • Useful for higher level monitoring and coordination of actions.

A Generic Process Schema iterate Ready Start Cancel Process Finish interrupt resume Done Suspend • Part of Conceptual Structure. • Generalizes over actions and events. Has internal state and models evolution of processes.

Aspects of (Climb) Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

About to + (Climb) (Prospective) Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Cancel + (Climb) Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Start + (Climb)-ING Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Be + (Climb)-ING (Progressive) Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Suspend (Climb)-ING Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Resumed + (Climb)-ING (Resumptive) Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Finish (End) + (Climb)-ING Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Have + (Climb)-ed (Perfect) Iterate Ready Start Process Finish Done resume interrupt Cancel Energy Ready Standing Suspend BINDINGS Hold Find hold Stabilize Pull(self) On top

Embedding: Has Started (to X) Ready Start Process Finish Done resume interrupt R S C P i F r D Suspend S X-Schema for X with bindings

Phasal Aspect Maps to the Controller Inceptive (start, begin) Ready Start Iterative (repeat) Iterate Process Finish interrupt resume Cancel Resumptive(resume) Done Suspend Completive (finish, end)

Embedding: About to start (X) Ready Start Process Finish resume interrupt R S C P i F r D S X-Schema for X with bindings Suspend Done

Embedding: Has Started (to X) Ready Start Process Finish Done resume interrupt R S C P i F r D Suspend S X-Schema for X with bindings

Begins and Ends • “This is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning. " – Speech given at the Lord Mayor's Luncheon, Mansion House, London, November 10, 1942. Winston Churchill

Embedding: It’s not this (the end) Ongoing Finish Done X-Schema for X with bindings

Embedding: It’s not this (beginning of the end) Ongoing Finish S R C P i Done F r D S X-Schema for X with bindings

Embedding: But this (The end of the beginning) Ready Start Process Finish Done resume interrupt R S C P i F r D Suspend S X-Schema for X with bindings

Inherent Aspect (Aksionsart) • Vendler-Dowty-Taylor (VDT) classification – Events and States – Events can be • Punctual or Durative • Atelic or Telic – States satisfy the downward entailment property • If a state holds in some interval, it holds in all subintervals of that interval.

Inherent Aspect • Much richer than traditional Linguistic Characterizations (VDT (durative/atomic, telic/atelic)) • Action patterns – one-shot, repeated, periodic, punctual – decomposition: concurrent, alternatives, sequential • Goal based schema enabling/disabling • Generic control features; – interruption, suspension, resumption • Resource usage

Basic Event X-schemas • • • State Event Transition Simple Event Simple Action Complex Event/Process Complex State

Aspectual Types

Other Transitions in the Controller may be coded • Lexical items may code interrupts – Stumble is an interrupt to an ongoing walk • A combination of grammatical and aktionsart may code of the controller phases – – – Ready to walk : Prospective Resuming his run: Resumptive Has been running: Embedded progressive About to Finish the painting: Embedded Completive. Canceling the meeting vs. Aborting the meeting.

Interaction of Aspect with Tense • Reichenbach’s system uses three pointers – Speech Time (S) – Reference Time (R) – Event Time (E) • Tense is a partial ordering relation between the pointers – Simple Past E < R, E < S – Perfect E < R < S

Viewpoint Aspect (Perfective/Imperfective)

Perfective/Imperfective Perfective Imperfective

Simulation and Reference Interval Perfective Imperfective

Levels of Granularity • Events can be construed at different levels of granularity based on various contextual factors. – In 1991, Mc. Enroe injured his knee while playing tennis. – This morning, I injured my knee while playing tennis. – He is coughing (normal time scale vs. slowmotion film time scale).

Summary of Aspect Results • Controller mediates between linguistic markings and individual event/verbal xschemas (Cogsci 99, Coling 2002) • Captures regular event structure; inspired by biological control theory • Flexible: specific events may require only a subset of controller; interaction of underlying xschemas, linguistic markers and hierarchical abstraction/ decomposition of controller accounts for wide range of aspectual phenomena. • Important aspectual distinctions, both traditional and novel, can be precisely specified in terms of the interaction of x-schemas with the controller (Cog. Sci 97, Cog. Sci 98, AAAI 99, IJCAI 99, Cog. Sci 04, Cog. Ling 2005): • • stative/dynamic, durative/punctual: natural in x-schemas telic processes: depletion of resources continuous processes: consumption of resources temporary/effortful states; habituals dynamic interactions with tense, nominals, temporal modifiers incorporation of world knowledge, pragmatics Ongoing Work: Simulation Semantics and Tense-Aspect (with Laura Michaelis)

Simulation hypothesis We understand utterances by mentally simulating their content. – Simulation exploits some of the same neural structures activated during performance, perception, imagining, memory… – Linguistic structure parameterizes the simulation. • Language gives us enough information to simulate

Simulation Semantics • BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE – Evidence for common mechanisms for recognition and action (mirror neurons) in the F 5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996) • IMPLEMENTATION: – x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x -schemas. State is completely distributed (a graph marking) over the network. • RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

A Precise Notion of Contingency Relations Activation: Executing one schema causes the enabling, start or continued execution of another schema. Concurrent and sequential activation. Inhibition: Inhibitory links prevent execution of the inhibited x-schema by activating an inhibitory arc. The model distinguishes between concurrent and sequential inhibition, mutual inhibition and aperiodicity. Modification: The modifying x-schema results in control transition of the modified xschema. The execution of the modifying x-schema could result in the interruption, termination, resumption of the modified x-schema.

General and Domain Knowledge • Conceptual Knowledge and Inference – – Embodied Language and Domain Independent Powerful General Inferences Ubiquitous in Language • Domain Specific Frames and Ontologies – Frame. Net, OWL ontologies • Metaphor links domain specific to general – E. g. , France slipped into recession.

Frames • Frames are conceptual structures that may be culture specific • Words evoke frames – The word “talk” evokes the Communication frame – The word buy (sell, pay) evoke the Commercial Transaction (CT) frame. – The words journey, set out, schedule, reach etc. evoke the Journey frame. • Frames have roles and constraints like schemas. – CT has roles vendor, goods, money, customer. • Words bind to frames by specifying binding patterns – Buyer binds to Customer, Vendor binds to Seller.

Event Frames Event frames have temporal structure that comprises of the controller event structure and generally have constraints on what precedes them, what happens during them, and what state the world is in once the event has been completed.

Sample Event Frame: Commercial Transaction Initial state: Vendor has Goods, wants Money Customer wants Goods, has Money Transition: Vendor transmits Goods to Customer transmits Money to Vendor Final state: Vendor has Money Customer has Goods

Sample Event Frame: Commercial Transaction Initial state: Vendor has Goods, wants Money Customer wants Goods, has Money Transition: Vendor transmits Goods to Customer transmits Money to Vendor Final state: Vendor has Money Customer has Goods (It’s a bit more complicated than that. )

Partial Wordlist for Commercial Transactions Verbs: pay, spend, cost, buy, sell, charge Nouns: cost, price, payment Adjectives: expensive, cheap

Meaning and Syntax § The various words that evoke this frame introduce the elements of the frame in different ways. § The identities of the buyer, seller, goods and money § Information expressed in sentences containing these words occurs in different places in the sentence depending on the word.

Language understanding: analysis & simulation construction WALKED form selff. phon [wakt] meaning : Walk-Action constraints selfm. time before Context. speech-time selfm. . aspect encapsulated “Harry walked into the cafe. ” Constructions Lexicon Utterance Analysis Process General Knowledge Semantic Specification Belief State CAFE Simulation