Natural Logic and Natural Language Inference Bill Mac
Natural Logic and Natural Language Inference Bill Mac. Cartney Stanford University / Google, Inc. 8 April 2011
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Two disclaimers • The work I present today isn’t exactly fresh • Essentially, it’s my dissertation work from 2009 • I hope it can usefully provide context for more recent work • I’m a computer scientist, not a semanticist or a logician • Consequently, I emphasize pragmatism over rigor 2
3 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Natural language inference (NLI) • Aka recognizing textual ‘entailment’ (RTE) • Does premise P justify an inference to hypothesis H? • An informal, intuitive notion of inference: not strict logic • Emphasis on variability of linguistic expression P H Every firm polled saw costs grow more than expected, Some even after adjusting for inflation. Every big company in the poll reported cost increases. Some • Necessary to goal of natural language understanding (NLU) • Can also enable semantic search, question answering, … no yes
4 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion NLI: a spectrum of approaches Solution? natural logic (this work) robust, but shallow lexical/ semantic overlap deep, but brittle Jijkoun & de Rijke 2005 patterned relation extraction FOL & theorem proving Romano et al. 2006 semantic graph matching Mac. Cartney et al. 2006 Hickl et al. 2006 Problem: imprecise easily confounded by negation, quantifiers, conditionals, factive & implicative verbs, etc. Bos & Markert 2006 Problem: hard to translate NL to FOL idioms, anaphora, ellipsis, intensionality, tense, aspect, vagueness, modals, indexicals, reciprocals, propositional attitudes, scope ambiguities, anaphoric adjectives, non-intersective adjectives, temporal & causal relations, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, comparatives, phrasal verbs, …
5 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion What is natural logic? ( natural deduction) • Characterizes valid patterns of inference via surface forms • precise, yet sidesteps difficulties of translating to FOL • A long history • traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, … • the term “natural logic” was introduced by Lakoff (1970) • van Benthem & Sánchez Valencia (1986 -91): monotonicity calculus • Nairn et al. (2006): an account of implicatives & factives • We introduce a new theory of natural logic • extends monotonicity calculus to account for negation & exclusion • incorporates elements of Nairn et al. ’s model of implicatives
6 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion ‘Entailment’ relations in past work X is a couch X is a crow X is a fish X is a hippo X is a man X is a sofa X is a bird X is a carp X is hungry X is a woman Yes 2 -way RTE 1, 2, 3 entailment 3 -way Yes Sánchez-Valencia non-entailment Unknown entailment Fra. Ca. S, PARC, RTE 4 containment No P=Q equivalence No compatibility P<Q forward entailment P>Q reverse entailment contradiction P#Q non-entailment
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion 16 elementary set relations Assign sets x, y to one of 16 relations, depending on emptiness or nonemptiness of each of four partitions y y x ? ? empty non-empty 7
8 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion 16 elementary set relations But 9 of 16 are degenerate: either x or y is either empty or universal. I. e. , they correspond to semantically vacuous expressions, which are rare outside logic textbooks. x^y We therefore focus on the remaining seven relations. x y x⊏y x‿ y x⊐y x|y x#y
9 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion The set of 7 basic entailment relations Venn symbol name example x y equivalence couch sofa x⊏y forward entailment crow ⊏ bird x⊐y reverse entailment European ⊐ French x^y negation human ^ nonhuman alternation cat | dog cover animal ‿ nonhuman independence hungry # hippo x|y x ‿ y x#y (strict) (exhaustive exclusion) (non-exhaustive exclusion) (exhaustive non-exclusion) Relations are defined for all semantic types: tiny ⊏ small, hover ⊏ fly, kick ⊏ strike, this morning ⊏ today, in Beijing ⊏ in China, everyone ⊏ someone, all ⊏ most ⊏ some
10 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Joining entailment relations ? x R y S z ⊏ ? ⊏ fish | human ^ nonhuman ⊐ ^ R ⋈ ⋈ ⋈ ⊏ ⊏ ⊐ ^ ⊐ R R R
11 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Some joins yield unions of relations! What is | | ? ⋈ x | y couch | table y | z table | sofa pistol | knife dog | cat knife | gun cat | terrier pistol rose | orchid woman | frog orchid | daisy frog | Eskimo rose | ⋈ | { , ⊏, ⊐, |, #} x ? z couch sofa ⊏ gun dog ⊐ terrier | daisy woman # Eskimo
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion The complete join table Of 49 join pairs, 32 yield relations in ; 17 yield unions Larger unions convey less information — limits power of inference In practice, any union which contains # can be approximated by # 12
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Lexical entailment relations ( x, e(x)) will depend on: 1. the lexical entailment relation generated by e: (e) atomic edit: DEL, INS, SUB 2. other properties of the context x in which e is applied compound expression entailment relation Example: suppose x is red car If e is SUB(car, convertible), then (e) is ⊐ If e is DEL(red), then (e) is ⊏ Crucially, (e) depends solely on lexical items in e, independent of context x But how are lexical entailment relations determined? 13
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Lexical entailment relations: SUBs (SUB(x, y)) = (x, y) For open-class terms, use lexical resource (e. g. Word. Net) for synonyms: sofa couch, forbid prohibit ⊏ for hypo-/hypernyms: crow ⊏ bird, frigid ⊏ cold, soar ⊏ rise | for antonyms and coordinate terms: hot | cold, cat | dog or | for proper nouns: USA United States, JFK | FDR # for most other pairs: hungry # hippo Closed-class terms may require special handling Quantifiers: all ⊏ some, some ^ no, no | all, at least 4 ‿ at most 6 See paper for discussion of pronouns, prepositions, … 14
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Lexical entailment relations: DEL & INS Generic (default) case: (DEL( • )) = ⊏, (INS( • )) = ⊐ • Examples: red car ⊏ car, sing ⊐ sing off-key • Even quite long phrases: car parked outside since last week ⊏ car • Applies to intersective modifiers, conjuncts, independent clauses, … • This heuristic underlies most approaches to RTE! • Does P subsume H? Deletions OK; insertions penalized. Special cases • Negation: didn’t sleep ^ did sleep • Implicatives & factives (e. g. refuse to, admit that): discussed later • Non-intersective adjectives: former spy | spy, alleged spy # spy • Auxiliaries etc. : is sleeping sleeps, did sleep slept 15
16 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion The impact of semantic composition How are entailment relations affected by semantic How is (x, y) projected by f? composition? ? @ f @ x y f [ @ means fn application The monotonicity calculus provides a partial answer If f has monotonicity… ⊏ ⊐ # UP ⊏ ⊐ # But how are other relations (|, ^, ‿ ) projected? ⊏ ⊐ # DOWN ⊐ ⊏ # ⊏ ⊐ # NON # # # ]
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A typology of projectivity Projectivity signatures: a generalization of monotonicity Eachclasses projectivity signature is a map ↦ In principle, 77 possible signatures, but few actually realized ⊏ ⊐ ^ | ‿ # negation ⊐ ⊏ ^ ‿ | # not happy didn’t kiss not ill not human not French not more than 4 isn’t swimming ⊐ ⊏ ^ not glad didn’t touch not seasick not nonhuman ‿ not German | not less than 6 # isn’t hungry 17
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A typology of projectivity Projectivity signatures: a generalization of monotonicity classes ↦ Each projectivity signature is a map In principle, 77 possible signatures, but few actually intersective negation modification realized ⊏ ⊐ ^ | ‿ # ⊐ ⊏ ^ ‿ | # ⊏ ⊐ ^ | ‿ # ⊏ ⊐ | | # # live human | live nonhuman French wine | Spanish wine metallic pipe # nonferrous pipe See my disseration for projectivity of various quantifiers, verbs 18
19 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Projecting through multiple levels Propagate entailment relation between atoms upward, according to projectivity class of each node on path to root nobody can enter without a shirt ⊏ nobody can enter without clothes ⊏ ⊐ @ @ @ nobody can without a shirt enter @ ⊐ @ ⊏ @ nobody can without clothes enter
20 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Implicatives & factives [Nairn et al. 06] 9 signatures, per implications (+, –, or o) in positive and negative contexts signature implicatives factives example +/– he managed to escape +/o he was forced to sell o/– he was permitted to live –/+ he forgot to pay –/o he refused to fight o/+ he hesitated to ask +/+ he admitted that he knew –/– he pretended he was sick o/o he wanted to fly
21 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Implicatives & factives We can specify relation generated by DEL or INS of each signature implicatives factives example (DEL) (INS) +/– he managed to escape he escaped +/o he was forced to sell ⊏ he sold ⊏ ⊐ o/– he was permitted to live he lived –/+ he forgot to pay ⊐ ^ he paid ⊐ ^ ⊏ ^ –/o he refused to fight | he fought | | o/+ he hesitated to ask ‿ he asked ‿ ‿ +/+ he admitted that he knew –/– he pretended he was sick ⊏ | ⊐ | o/o he wanted to fly # he flew # # Factives Room fornotvariation fully explained: w. r. t. he didn’t complementizers, admithesitate thatpassivation, he knew etc. | |he Some more intuitive wheninfinitives, negated: he didn’t to ask hedidn’tknow ask
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Putting it all together 1. Find a sequence of edits e 1, …, en which transforms p into h. Define x 0 = p, xn = h, and xi = ei(xi– 1) for i [1, n]. 2. For each atomic edit ei: 1. Determine the lexical entailment relation (ei). 2. Project (ei) upward through the semantic composition tree of expression xi– 1 to find the atomic entailment relation (xi– 1, xi) 3. Join atomic entailment relations across the sequence of edits: (p, h) = (x 0, xn) = (x 0, x 1) ⋈ … ⋈ (xi– 1, xi) ⋈ … ⋈ (xn– 1, xn) Limitations: need to find appropriate edit sequence connecting p and h; tendency of ⋈ operation toward less-informative entailment relations; lack of general mechanism for combining multiple premises Less deductive power than FOL. Can’t handle e. g. de Morgan’s Laws. 22
23 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion An example The doctor didn’t hesitate to recommend Prozac. The doctor recommended medication. P H i ei xi lex yes atom join ‿ | | ^ ^ ⊏ ⊏ yes The doctor didn’t hesitate to recommend Prozac. 1 DEL(hesitate to) The doctor didn’t recommend Prozac. 2 DEL(didn’t) The doctor recommended Prozac. 3 SUB(Prozac, medication) The doctor recommended medication.
24 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Different edit orders? i ei lex atom join 1 DEL(hesitate to) ‿ | | 2 DEL(didn’t) ^ ^ ⊏ 2 SUB(Prozac, medication) ⊏ ⊐ | 3 SUB(Prozac, medication) ⊏ ⊏ ⊏ 3 DEL(didn’t) ^ ^ ⊏ i ei lex atom join 1 DEL(didn’t) ^ ^ ^ 2 DEL(hesitate to) ‿ ‿ ⊏ 2 SUB(Prozac, medication) ⊏ ⊐ | 3 SUB(Prozac, medication) ⊏ ⊏ ⊏ 3 DEL(hesitate to) ‿ ‿ ⊏ i ei lex atom join 1 SUB(Prozac, medication) ⊏ ⊏ ⊏ 2 DEL(hesitate to) ‿ | | 2 DEL(didn’t) ^ ^ | 3 DEL(didn’t) ^ ^ ⊏ 3 DEL(hesitate to) ‿ ‿ ⊏ Intermediate steps may vary; final result is typically (though not necessarily) the same
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Implementation & evaluation The Nat. Log system: an implementation of this model in code For implementation details, see [Mac. Cartney & Manning 2008] Evaluation on Fra. Ca. S test suite 183 NLI problems, nine sections, three-way classification Accuracy 70% overall; 87% on “relevant” sections (60% coverage) Precision 89% overall: rarely predicts entailment wrongly Evaluation on RTE 3 test suite Longer, more natural premises; greater diversity of inference types Nat. Log alone has mediocre accuracy (59%) but good precision Hybridization with broad-coverage RTE system yields gains of 25
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusions Natural logic is not a universal solution for NLI Many types of inference not amenable to natural logic approach Our inference method faces many limitations on deductive power More work to be done in fleshing out our account Establishing projectivity signatures for more quantifiers, verbs, etc. Better incorporating presuppositions But, our model of natural logic fills an important niche : -) Precise reasoning on negation, antonymy, quantifiers, implicatives, … Sidesteps the myriad difficulties of full semantic interpretation Practical value demonstrated on Fra. Ca. S and RTE 3 test suites Thanks! Questions? 26
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Backup slides follow 27
28 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion An example involving exclusion Stimpy is a cat. Stimpy is not a poodle. P H i ei xi yes lex atom join | | | ^ ^ ⊏ ⊐ ⊏ ⊏ yes Stimpy is a cat. 1 SUB(cat, dog) Stimpy is a dog. 2 INS(not) Stimpy is not a dog. 3 SUB(dog, poodle) Stimpy is not a poodle.
29 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion An example involving an implicative We were not permitted to smoke. We smoked Cuban cigars. P H i ei xi no lex atom join ⊐ ⊏ ⊏ ^ ^ | ⊐ ⊐ | We were not permitted to smoke. 1 DEL(permitted to) We did not smoke. 2 DEL(not) We smoked. 3 INS(Cuban cigars) We smoked Cuban cigars. no
30 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion de Morgan’s Laws for quantifiers Not all birds fly. Some birds do not fly. P H i ei xi yes lex atom join ^ ^ ^ ⊏ ⊏ ‿ ^ ‿ Not all birds fly. 1 DEL(not) All birds fly. 2 SUB(all, some) Some birds fly. 3 INS(not) Some birds do not fly. ⊏⊐‿ # wtf? ?
31 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion de Morgan’s Laws for quantifiers (2) Not all birds fly. Some birds do not fly. P H i ei xi yes lex atom join ^ ^ | ⊐ ⊏ ⊏ Not all birds fly. 1 DEL(not) All birds fly. 2 INS(not) All birds do not fly. 3 SUB(all, some) Some birds do not fly. ⊏⊐‿ # wtf? ?
32 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A more complex example Jimmy Dean refused to move without blue jeans. James Dean didn’t dance without pants. yes P H i ei lex atom join SUB(Jimmy Dean, James Dean) 2 DEL(refuse to) | | | 3 INS(did) INS(n’t) ^ | 4 ^ ⊏ 5 SUB(move, dance) ⊐ ⊏ ⊏ 6 DEL(blue) ⊏ ⊏ ⊏ 7 SUB(jeans, pants) ⊏ ⊏ ⊏ 1
33 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A more complex example (2) Jimmy Dean refused to move without blue jeans. James Dean didn’t dance without pants. yes P H i ei lex atom join ^ ^ ^ 1 INS(did) 2 INS(n’t) 3 DEL(blue) ⊏ ⊐ | 4 SUB(jeans, pants) ⊏ ⊐ | ⊐ ⊐ | 5 SUB(move, dance) 6 DEL(refuse to) | ‿ ⊏ 7 SUB(Jimmy Dean, James Dean) ⊏
34 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A more complex example (3) Jimmy Dean refused to move without blue jeans. James Dean didn’t dance without pants. yes P H i ei lex atom join | | 1 INS(did) 2 INS(n’t) ^ 6 DEL(refuse to) | | ⊏⊐|# 3 DEL(blue) ⊏ ⊏ • 4 SUB(jeans, pants) ⊏ ⊏ • ⊐ ⊐ • 5 7 SUB(move, dance) SUB(Jimmy Dean, James Dean)
- Slides: 34