Temporal Relations in Visual Semantics of Verbs Minhua

Temporal Relations in Visual Semantics of Verbs Minhua Eunice Ma and Paul Mc Kevitt School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee Derry/Londonderry, N. Ireland

Background: CONFUCIUS (intelligent storytelling system) Storywriter /playwright nguage a l l a r u t a n n story i movie/drama script CONFUCIUS tailored m enu for sc ript input speec h (dialo gue) 3 D animation dio no au h c e e p n-s User /story listener w To interpret natural language stories and to extract conceptual semantics from natural language w To generate 3 D animation and virtual worlds automatically from natural language w To integrate 3 D animation with speech and non-speech audio for presenting multimodal stories AICS 2003, Dublin, Ireland

Previous research w Temporal relations n n n Allen’s interval relations Application in story-based interactive systems Temporal relations in technical orders domain (Badler et al. , 1997) w Related research in NLP n n n Sentence level temporal analysis Lexical vs. post-lexical temporal relations Lexical semantics AICS 2003, Dublin, Ireland

Allen’s interval relations AICS 2003, Dublin, Ireland

NLP in CONFUCIUS Preprocessing Part-of-speech tagger Connexor FDG parser Syntactic parser Semanti c inferenc Word. Net e LCS database Disambiguati on FEATURES LEXICON & MORPHOLOGICAL RULES Morphologi cal parser Coreferen ce resolution Temporal reasoning Post-lexical Lexical temporal relations AICS 2003, Dublin, Ireland

Verb entailments Verb entailment: fixed truth relation between verbs with entailment given by part of lexical meaning, i. e. one verb entails another The implication logic relationship: if p then q (p q) AICS 2003, Dublin, Ireland

Troponym w Elaborates manner of base verb (Fellbaum, 1998) w Examples: “trot”-“walk” (fast), “gulp”-“eat” (quickly) EVENT … go (move) cause other action predicates … walk run climb jump manner-of-motion verbs … limp stride trot swagger AICS 2003, Dublin, Ireland

Temporal relations in verb entailment {p, m, o, s, f-1, ≡} may also represent temporal relation between pair of cognate verbs and state of corresponding adjectives e. g. shorten-short, beautify-beautiful, clarify-clear AICS 2003, Dublin, Ireland

Representing procedural events Arguments of EVENT [EVENT agent: theme: space/time: manner: instrument: precondition: subactivities: result: ] Relationship between definiendum verb and its subactivities act(): subact 1(), …… subacti(), ……. subacti R act, i N, R {d, s, f, } eat. Out(): book. ASeat() , go. To. Restaurant() , order. Dishes() , eat() , pay() , leave(). a. Original definition eat. Out(): book. ASeat() {p} go. To. Restaurant(){p, m} order. Dishes() {p} eat() {p, m} pay() {p, m} leave(). b. “eat. Out” in restaurant eat. Out(): book. ASeat() {p} go. To. Restaurant(){p, m} order. Dishes() {p} eat() {p, p-1, m} pay() {p, m} leave(). c. “eat. Out” in restaurant/ fast food shop eat. Out(): [book. ASeat() {p}] go. To. Restaurant(){p, m} order. Dishes() {p} eat() {p, p-1, m} pay() {p, m} leave(). d. Optional subactivities AICS 2003, Dublin, Ireland

Comparison with Badler’s temporal constraints (technical orders domain) Interval relations Sequential {p, m} Parallel {s, s-1, } Jointly parallel (act 1 {s, s-1, } act 2) {p, m} act 3 Independently parallel {f, f-1, } While parallel act_domt {s-1, f-1, } act_indomt • compositional (e. g. jointly parallel); all 5 constraints are disjunctions of several interval relations • consider other factors such as dominancy of action (e. g. while parallel) • domain-specific AICS 2003, Dublin, Ireland

Comparison with Badler’s temporal constraints (technical orders domain) Interval relations Sequential {p, m} Parallel {s, s-1, } Jointly parallel (act 1 {s, s-1, } act 2) {p, m} act 3 Independently parallel {f, f-1, } While parallel act_domt {s-1, f-1, } act_indomt • compositional (e. g. jointly parallel); all 5 constraints are disjunctions of several interval relations • consider other factors such as dominancy of action (e. g. while parallel) • domain-specific AICS 2003, Dublin, Ireland

Comparison with Badler’s temporal constraints (technical orders domain) Interval relations Sequential {p, m} Parallel {s, s-1, } Jointly parallel (act 1 {s, s-1, } act 2) {p, m} act 3 Independently parallel {f, f-1, } While parallel act_domt {s-1, f-1, } act_indomt • compositional (e. g. jointly parallel); all 5 constraints are disjunctions of several interval relations • consider other factors such as dominancy of action (e. g. while parallel) • domain-specific AICS 2003, Dublin, Ireland

Comparison with Badler’s temporal constraints (technical orders domain) Interval relations Sequential {p, m} Parallel {s, s-1, } Jointly parallel (act 1 {s, s-1, } act 2) {p, m} act 3 Independently parallel {f, f-1, } While parallel act_domt {s-1, f-1, } act_indomt • compositional (e. g. jointly parallel); all 5 constraints are disjunctions of several interval relations • consider other factors such as dominancy of action (e. g. while parallel) • domain-specific AICS 2003, Dublin, Ireland

Comparison with Badler’s temporal constraints (technical orders domain) Interval relations Sequential {p, m} Parallel {s, s-1, } Jointly parallel (act 1 {s, s-1, } act 2) {p, m} act 3 Independently parallel {f, f-1, } While parallel act_domt {s-1, f-1, } act_indomt • compositional (e. g. jointly parallel); all 5 constraints are disjunctions of several interval relations • consider other factors such as dominancy of action (e. g. while parallel) • domain-specific AICS 2003, Dublin, Ireland

Achievement vs. accomplishment events Achievement events (Vendler, 1967): e. g. “find”, “arrive”, “die” n punctual events occuring at single moment n definite time instants n never hold over intervals Why use interval relations instead of point-based relations? n Pragmatic reasons (Verkuyl, 1993) n Ontological reasons (Pinon, 1997) n Practical reason for language visualisation • • achievement events depend on existence of context + visual definitions → intervals find(): search(), eyes. Fixed. On(). arrive(): go(), stop. At. Destination(). AICS 2003, Dublin, Ireland

Temporal relations of lexical causatives w Visual definitions of causative verbs (e. g. “kill”) must subsume result states (stative verbs) (e. g. “die”) w Represent distinction between launching causatives: causation of inception of motion entraining causatives: continuous causation of motion disjunction set of interval relations between cause and effect adequate to define difference: {s, p, m, o} (launching) {≡, f-1} (entraining) AICS 2003, Dublin, Ireland

Lexical and post-lexical repetition w Post-lexical level repetition e. g. “Roses come into bloom once a year. ” “I visit the school every day. ” or marked by “again", "continues to", "a second time” w Lexical level repetition n n Represent periodical repetition of subactivities walk(): - [step()]R. hammer(): - [hit()]R. Morphological prefix "re-" AICS 2003, Dublin, Ireland

Categories of action verb 2. 2. 1. Action verbs 2. 2. 1. 1. Movement or partial movement 2. 2. 1. 1. 1. Biped kinematics, e. g. go, walk, jump, swim, climb 2. 2. 1. 1. 2. Face expressions, e. g. laugh, angry involve speech modality 2. 2. 1. 1. 3. Lip movement, e. g. speak, say, sing, tell 2. 2. 1. 2. Lexical causatives 2. 2. 1. Concerning single object, e. g. push, kick, bring, open 2. 2. 1. 2. 2. Concerning multiple objects 2. 2. 1. Bitransitive verbs, e. g. give, sell, show 2. 2. 1. 2. 2. 2. Transitive verbs with object & implicit instrument/goal/theme, e. g. cut, write, butter, pocket 2. 2. 1. 3. Verbs without distinct visualization when out of context 2. 2. 1. 3. 1. trying verbs: try, attempt, succeed, manage 2. 2. 1. 3. 2. helping verbs: help, assist 2. 2. 1. 3. 3. letting verbs: allow, let, permit 2. 2. 1. 3. 4. create/destroy verbs: build, create, assemble, construct, break, destroy 2. 2. 1. 3. 5. verbs whose visualization depends on their objects, e. g. play (harmonica/football), make (the bed/trouble/a phone call), fix (a drink/a lock) 2. 2. 1. 4. High level behaviours events) AICS (routine 2003, Dublin, Ireland

Lexical Visual Semantic Representation w Lexical Visual Semantic Representation (LVSR): necessary semantic representation between 3 D model and language syntax w LVSR based on Jackendoff’s LCS adapted to task of language visualization (enhancement with Schank’s scripts) w Interval relations represent temporal relationship between subactivities of complex actions in LVSR w e. g. “The waiter approached me: ‘Can I help you? Sir. ’” 3 D animation “John walked towards the house. ” 3 D animation “Nancy ran across the field. ” 3 D animation AICS 2003, Dublin, Ireland

Conclusion ♦ Temporal relation is a crucial issue in modelling action verbs, their procedures, contexts, presupposed and result states ♦ Temporal relation within verb semantics (lexical level) ♦ Semantic representation of verbs with temporal information based on Allen’s interval logic AICS 2003, Dublin, Ireland

Future work w Quantitative factor w Action composition for simultaneous activities w Verbs concerning multiple characters’ synchronization & coordination Character can start a task when another signals pre-conditions are ready n n Two or more characters cooperate in shared task AICS 2003, Dublin, Ireland