Visual Semantics and Ontology of Eventive Verbs Minhua

Visual Semantics and Ontology of Eventive Verbs Minhua Eunice Ma and Paul Mc Kevitt School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee Derry/Londonderry, N. Ireland

Outline w Background: CONFUCIUS w Previous verb taxonomies w Visual semantics & verb classes w CONFUCIUS’ ontology of verbs w Current status of implementation w Relation to other work w Conclusion & future work IJCNLP-04 Sanya, China, 22 Mar 2004

Architecture of CONFUCIUS Natural language sentences Surface transformer Media allocator Prefabricated objects (knowledge base) Language knowledge mapping 3 D authoring tools, existing 3 D models & character models Visual knowledge (3 D graphic library) LCS lexicon Natural Word. Net Language Processing Text To Speech Sound effects semantic representations visual knowledge Animation engine Synchronising & fusion 3 D world with audio in VRML IJCNLP-04 Sanya, China, 22 Mar 2004

NLP in CONFUCIUS Preprocessing Part-of-speech tagger Connexor FDG parser Syntactic parser Semanti c inferenc Word. Net e LCS database Disambiguati on FEATURES Morphologi cal parser Temporal Coreferen reasoning ce resolution Post-lexical Lexical temporal relations IJCNLP-04 Sanya, China, 22 Mar 2004

Previous verb taxonomies w Grammatical categorisation & valency w Thematic roles (Fillmore, 1968; Jackendoff, 1990; Halliday, 1985; Dowty, 1991) w Aspectual classes (Vendler, 1967; Stede, 1996) w Semantic verb classes in Word. Net (Fellbaum, 1998) w Levin’s (1993) verb classes w Dimension of causation (Asher & Lascarides, 1995) IJCNLP-04 Sanya, China, 22 Mar 2004

Grammatical categorisation & valency w Subcategorisation description of verb categories in LDOCE (Longman Dictionary of Contemporary English) n n n D – ditransitive I – intransitive L – linking verb with complement T 1 – transitive verb with NP object T 3 – transitive verb with infinitival clause as object IJCNLP-04 Sanya, China, 22 Mar 2004

Grammatical categorisation & valency w Subcategorisation description of verb categories in LDOCE (Longman Dictionary of Contemporary English) w Syntactic valency n Obligatory valency fillers (complements) e. g. subject, object n Optional valency fillers (adjuncts) e. g. temporal, locational adjuncts w Semantic valency (Leech, 1981) IJCNLP-04 Sanya, China, 22 Mar 2004

Thematic roles w Other names: theta-role, case role, deep grammatical function, transitivity role, valency role, case frame w Extend syntactic analysis into semantic domain to capture roles of participants surface case (nominative, accusative) surface function (subject, object) Thematic roles (e. g. agent, patient/theme, instrument, source, goal, place) w Classifying verbs based on thematic roles (Dixon, 1991) IJCNLP-04 Sanya, China, 22 Mar 2004

Aspectual (temporal) classification w Vendler’s (1967) verb classes n n w w activities: run, swim, sleep, cry statives: love, hate, know achievements: arrive, win, find, die accomplishments: build (a house), write (a book) Stede’s (1996) MOOSE ontology Formal ontologies DOLCE, SUMO, and CYC assume traditional aspectual (temporal) classification for events IJCNLP-04 Sanya, China, 22 Mar 2004

Aspectual (temporal) classification w w Vendler’s (1967) verb classes Stede’s (1996) ontology of MOOSE situation state activity event protracted moment culmination transition activity protracted moment culmination w Formal ontologies DOLCE, SUMO, and CYC assume traditional aspectual (temporal) classification for events IJCNLP-04 Sanya, China, 22 Mar 2004

Aspectual (temporal) classification w w Vendler’s (1967) verb classes Stede’s (1996) ontology of MOOSE situation state activity event protracted moment culmination transition activity protracted moment culmination w Formal ontologies DOLCE, SUMO, and CYC assume traditional aspectual (temporal) classification for events IJCNLP-04 Sanya, China, 22 Mar 2004

Semantic verb classes in Word. Net Lexicographer file Contents verb. body verb. change verb. cognition verb. communication verb. competition verb. consumption verb. contact verb. creation verb. emotion verb. perception verb. possession verb. social verb. stative verb. weather grooming, dressing, bodily care size, temperature change thinking, judging, doubting telling, asking, ordering, singing fighting, athletic activities eating and drinking touching, hitting, tying, digging sewing, baking, painting feeling walking, flying, swimming seeing, hearing, feeling buying, selling, owning political/social activities being, having, spatial relations raining, snowing, thundering IJCNLP-04 Sanya, China, 22 Mar 2004 w Taxonomic approach based on pure lexical semantics w Reveal semantic organisation of lexicon in terms of lexical & semantic relations w Top nodes of Word. Net’s verb file

Levin’s (1993) verb classes w Theoretic ground -- semantic/syntactic correlations: verbs with similar meaning (identical LCSs in terms of specific meaning components) show same syntactic behaviors Verbs of motion Verbs of inherently directed motion: arrive, come, enter Leave verbs: leave, abandon, desert Manner of motion verbs: roll, run, sneak, waddle Verbs of motion using a vehicle: bike, drive, fly Chase verbs: chase, follow, track Accompany verbs: accompany, escort, guide Waltz verbs: clog, polka, waltz IJCNLP-04 Sanya, China, 22 Mar 2004

Dimension of causation w Asher and Lascarides’ (1995) dimension of causation-change w causation and change are specified along four dimensions: locative, formal, matter, intentional cause locative loc-cause fml-cause mtr-cause sub of put sub of build sub of paint formal intent-cause sub of amuse matter loc-change obj of put intentional fml-change obj of build IJCNLP-04 Sanya, China, 22 Mar 2004 change intent-change mtr-change obj of amuse obj of paint

Visual semantics & verb classes Visual factors concerning verb categorisation w Visual valency w Somatotopic factors in visualisation w Level-of-detail of visual information Verbs belonging to same class in the classification w Visual “synonyms” w Substitutable in same set of animation keyframes Visualisation of action verbs is effective evaluation of the classification IJCNLP-04 Sanya, China, 22 Mar 2004

Visual valency w Capacity of verb to take specific number and type of visual arguments in language visualisation (3 D animation) w valency filler -- visual role w 2 types of visual roles requiring different processes in visualisation n n human (biped articulated animate entity) object (inanimate entity) w Visual valency overlaps with syntactic & semantic valency w Visual modality requires more obligatory roles than surface grammar or semantics IJCNLP-04 Sanya, China, 22 Mar 2004

Somatotopic effectors of action verbs w Theoretical ground: execution/perception/visualisation of action verbs produced by same somatotopic effector activate same parts of cortex w Distinguish facial expression (e. g. lip movement) & body posture (arm/leg/torso) in our ontological system facial expression – sing, laugh Leg – run, kick body posture Arm – wave, put torso – bow w Further divisions like distinction between upper/lower arm, hands, & even fingers are possible IJCNLP-04 Sanya, China, 22 Mar 2004

Level-Of-Detail (LOD) basic-level verbs & troponyms EVENT … go cause event level verbs … walk climb limp stride trot swagger Levels run jump manner level verbs jog romp skip bounce hop Verbs troponym level verbs Basic-level verb for. . . event level go, move atomic object manner level walk, jump human/non-atomic object troponym level limp, stride human IJCNLP-04 Sanya, China, 22 Mar 2004

CONFUCIUS’ verb taxonomy 2. 2. 1. Action verbs 2. 2. 1. 1. One visual valency (the role is a human, (partial) movement) 2. 2. 1. 1. 1. Biped kinematics: arm actions (wave, scratch), leg actions (walk, jump, kick), torso actions (bow), combined actions (climb) 2. 2. 1. 1. 2. Facial expressions & lip movement, e. g. laugh, fear, say, sing, order 2. 2. 1. 2. Two visual valency (at least one role is human) 2. 2. 1. One human and one object (vt. or vi. +instrument) e. g. throw, push, kick, open, eat, drink, bake, trolley 2. 2. 1. 2. 2. Two humans, e. g. fight, chase, guide 2. 2. 1. 3. Visual valency ≥ 3 (at least one role is human) 2. 2. 1. 3. 1. Two humans and one object (inc. ditransitive verbs), e. g. give, show 2. 2. 1. 3. 2. One human and 2+ objects (vt. + object + implicit instr. /goal/theme) e. g. cut, write, butter, pocket, dig, cook 2. 2. 1. 4. Verbs without distinct visualisation when out of context: verbs of trying, helping, letting, creating/destroying 2. 2. 1. 5. High level behaviours (routine events), political and social activities e. g. interview, eat out (go to restaurant), go shopping IJCNLP-04 Sanya, China, 22 Mar 2004

Text-to-Animation of single sentences w Collision detection example (contact verbs: hit, collide, scratch, touch), no human role involved “The car collided with a wall. ” n using Parallel. Graphics’ VRML extension--object-to-object collision n non-speech sound effects w H-Anim examples: action verbs 3 visual valency verbs “John gave Nancy a loaf of bread. ” “John put a cup of coffee on the table. ” n H-Anim Site node n locative tags of object (on_table tag for table object) 2 visual valency verbs “John pushed the door. ” “John ate the bread. ” “Nancy sat on the chair. ” For more demos, please visit http: //www. infm. ulst. ac. uk/~eunice/3 D_anim. html IJCNLP-04 Sanya, China, 22 Mar 2004

Relation to other work w Categorise verbs from visual semantics perspective w Language visualisation in CONFUCIUS provides independent criteria for identifying classes of verbs sharing certain aspects of meaning, i. e. semantic/visual correlations w Relation to Levin’s verb classes: “Carol cut the whole wheat bread. ” “Whole wheat bread cuts easily. ” 2. 2. 1. 3. 2, visual valency=3 2. 1. 2, visual valency=2 Verbs of cutting 1 to N “Nancy brought the book to John. ” Verbs of sending & carrying 2. 2. 1. 3. 1 “Nancy gave the book to John. ” Verbs of change of possession visual valency=3 N to 1 IJCNLP-04 Sanya, China, 22 Mar 2004

Conclusion & future work w Categorise verbs from visual semantic perspective w Provides independent criteria for identifying classes of verbs based on semantic/visual correlations w Visual semantic analysis of eventive verbs revealed striking influences in taxonomic verb tree w Various criteria ranging from visual valency, somatotopic effectors, to LOD are proposed w Evaluation issues using automatic animation generation & psychological experiments w Discourse level interpretation IJCNLP-04 Sanya, China, 22 Mar 2004