Recognizing Partial Textual Entailment Torsten Zesch Iryna Gurevych

Recognizing Partial Textual Entailment Torsten Zesch Iryna Gurevych Ubiquitous Knowledge Processing Lab Technische Universität Darmstadt Omer Levy Ido Dagan Natural Language Processing Lab Bar-Ilan University 1

Textual Entailment 2

T: muscles move bones 3

T: muscles move bones H: muscles generate movement 4

T: muscles move bones H: muscles generate movement 5

T: muscles move bones ? H: the muscles’ main job is to move bones 6

T: muscles move bones H: the muscles’ main job is to move bones 7

Complete Textual Entailment 8

Complete Textual Entailment 9

. Complete Partial Textual Entailment. 10

. Complete Partial Textual Entailment. (Nielsen et al, 2009) 11

T: muscles move bones H: the muscles’ main job is to move bones 12

T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones 13

T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones 14

T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones n e d Stu e s n o p s t Re is s y l Ana 15

T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones is s n y l o i a t n c n A a o i r t e t s a x z n E i r o n a p s io m t e a R m t u m r S n o e f In Stud 16

Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones 17

Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones) 18

Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones) 19

Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones) 20

Facet: a pair of terms and their implied semantic relation. 21

(move, bones) Facet: a pair of terms and their implied semantic relation. 22

(move, bones) Facet: a pair of terms and their implied semantic relation. theme … is to move bones 23

Recognizing Faceted Entailment 24

Recognizing Faceted Entailment T: muscles move bones H: the muscles’ main job is to move bones 25

Recognizing Faceted Entailment T: muscles move bones H: the muscles’ main job is to move bones 26

Recognizing Faceted Entailment T: muscles move bones H: the muscles’ main job is to move bones 27

Recognizing Faceted Entailment T: muscles move bones H: the muscles’ main job is to move bones 28

What did we do? 29

Approach: Leverage Existing Methods 30

Approach: Leverage Existing Methods 31

Approach: Leverage Existing Methods 32

Approach: Leverage Existing Methods 33

Approach: Leverage Existing Methods 34

Approach: Leverage Existing Methods 35

Approach: Leverage Existing Methods penicillin cures pneumonia is treated by penicillin 36

Bar-Ilan University Textual Entailment Engine (Stern and Dagan, 2011) 37

BIUTEE 38

BIUTEE T H cures penicillin pneumonia 39

BIUTEE T H cures penicillin treated pneumonia is by penicillin 40

BIUTEE T H cures penicillin treated pneumonia is by penicillin 41

cures penicillin pneumonia 42

cures penicillin pneumonia treats penicillin pneumonia 43

cures penicillin pneumonia treated pneumonia is by penicillin 44

Facets to Dependencies treated pneumonia is by penicillin 45

Facets to Dependencies treated pneumonia is by penicillin 46

Decision Mechanisms Baseline Exact Match Base. Lex Exact Match OR Lexical Inference Base. Syn Exact Match OR Syntactic Inference Lexical Inference Majority Exact Match OR AND Syntactic Inference 47

Decision Mechanisms Baseline Exact Match Base. Lex Exact Match OR Lexical Inference Base. Syn Exact Match OR Syntactic Inference Lexical Inference Majority Exact Match OR AND Syntactic Inference 48

Decision Mechanisms Baseline Exact Match Base. Lex Exact Match OR Word. Net Base. Syn Exact Match OR BIUTEE Lexical Inference Majority Exact Match OR AND Syntactic Inference 49

Decision Mechanisms Baseline Exact Match Base. Lex Exact Match OR Word. Net Base. Syn Exact Match OR BIUTEE Word. Net Majority Exact Match OR AND BIUTEE 50

Sem. Eval 2013 Task 7 51

Dataset Sci. Ents. Bank (Djikovska et al, 2012) 15 Domains 5, 106 Student Responses 16, 263 Facets 58

Dataset Sci. Ents. Bank (Djikovska et al, 2012) 15 Domains 5, 106 Student Responses 16, 263 Facets 59

Dataset Sci. Ents. Bank (Djikovska et al, 2012) 15 Domains 5, 106 Student Responses 16, 263 Facets 60

Dataset Sci. Ents. Bank (Djikovska et al, 2012) 15 Domains 13, 145 Train Facets 16, 263 Test Facets 61

Dataset Sci. Ents. Bank (Djikovska et al, 2012) 15 Domains 13, 145 Train Facets 16, 263 Test Facets 62

Scenarios Unseen Answers 9% Unseen Questions 12% Unseen Domains 79% 63

Scenarios Unseen Answers 9% Unseen Questions 12% Unseen Domains 79% 64

Scenarios Unseen Answers 9% Unseen Questions 12% Unseen Domains 79% 65

Scenarios Unseen Answers 9% Unseen Questions 12% Unseen Domains 79% 66

Results 67

Performance (Weighted F 1) 1 0, 8 Baseline Base. Lex Base. Syn Majority 0, 6 0, 4 0, 2 0 Unseen Answers Unseen Questions Unseen Domains 68

Summary 69

Problem: Need fine-grained entailment Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement 70

Problem: Need fine-grained entailment Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement 71

Problem: Need fine-grained entailment Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement 72

Problem: Need fine-grained entailment Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement 73

Partial Textual Entailment is interesting 76

Partial Textual Entailment is useful 77

Partial Textual Entailment is practical 78

Partial Textual Entailment is interesting 79

Questions? 80