LREC 2008 Marrakech 28 May 2008 FATE a

LREC 2008 , Marrakech , 28 May 2008 FATE: a Frame. Net Annotated corpus for Textual Entailment Marco Pennacchiotti, Aljoscha Burchardt Computerlinguistik Saarland University, Germany SALSA II - The Saarbrücken Lexical Semantics Acquisition Project

Summary • Frame. Net and Textual Entailment • FATE annotation schema • Annotation examples and statistics • Conclusions 28/05/2008 FATE - Marco Pennacchiotti 2 / 17

Frame Semantics [Fillmore 1976, 2003] • Frame: conceptual structure modeling a prototypical situation • Frame Elements (FE): participants of the situation • Frame Evoking elements (FEE): predicates evoking the situation Predicate-argument level normalizations “Evelyn spoke about her past” “Evelyn’s statement about her past” STATEMENT(SPEAKER: Evelyn; TOPIC: her past • Frame. Net Berkeley Project 1 – Database of frames for the core lexicon of English – 800 frames, 10. 000 lemmas, 135. 000 annotated sentences 28/05/2008 FATE - Marco Pennacchiotti (1) http: //framenet. icsi. berkeley. edu 3 / 17

Textual Entailment (TE) Given two text fragments, the Text T and the Hypothesis H, T entails H if the meaning of H can be inferred from the meaning of T, as would typically interpreted by people [Dagan 2005] T: “Yahoo has recently acquired Overture” H: “Yahoo owns Overture” T H • Recognizing Textual Entailment (RTE) – recognize if entailment holds for a given (T, H) pair – Models core inferences of many NLP applications (QA, IE, MT, …) • RTE Challenges [Dagan et al. , 2005 ; Giampiccolo et al. , 2007] – Compare systems for RTE – Corpus: 800 training pairs, 800 test pairs, evenly split in + and - pairs 28/05/2008 FATE - Marco Pennacchiotti 4 / 17

Predicate-argument and RTE • Predicate-level inference plays a relevant role in TE (20% of positive examples in RTE-2 [Garoufi, 2007] ) T An avalanche has struck a popular skiing resort in Austria, killing at least 11 people. H Humans died in an avalanche. DEATH(PROTAGONIST: 11 people / humans ; CAUSE: avalanche / avalanche ) • Implementation gap : • • • 28/05/2008 [Burchardt et al. , 2007] : Frame. Net system comparable to lexical overlap [Hickl et al. , 2006] : Prop. Bank-based features are not effective [Rana et al. , 2005]: DIRT paraphrase repository does not help FATE - Marco Pennacchiotti 5 / 17

FATE corpus FATE: a manually frame-annotated Textual Entailment corpus, to study the role of frame semantics in RTE • • • Reference corpus Frame resource Corpus Format : RTE-2 test set, 800 pairs, 29, 000 tokens : Frame. Net version 1. 3 : SALSA/TIGER XML [Burchardt et al. , 2006] • Pre-processing : annotation on top of Collins parser syntactic analysis : T and H are randomly reordered to avoid biases • Annotation : performed by one highly experienced annotator : inter-annotator agreement over 5% of the corpus – – – FEE-agreement : 82% Frame-agreement: 88% Role-agreement: 91% : annotation carried out using the SALTO tool 1 (1) http: //www. coli. uni-saarland. de/projects/salsa/salto/doc 28/05/2008 FATE - Marco Pennacchiotti 6 / 17

FATE annotation process: an example Collins synt. an. full-text annotation (all words considered) [Ruppenhofer, 2007] 28/05/2008 FATE - Marco Pennacchiotti 7 / 17

FATE annotation process: an example frame Collins synt. an. FEE 28/05/2008 FATE - Marco Pennacchiotti 8 / 17

FATE annotation process: an example frame FE Collins synt. an. FEE FE filler Maximization principle: chose the largest constituent possible when annotating 28/05/2008 FATE - Marco Pennacchiotti 9 / 17

Annotation Schema Relevance Principle • Intuition: annotate as FEE only those words evoking a relevant situation (frame) in the sentence at hand – Very intuitive flavor, but high agreement: 83% on a pilot set of 15 sentences LEADERSHIP PERPETRATO R DETAIN PLACE PEOPLE KIDNAPPING VICTIM “Authorities in Brazil hold 200 people as hostage” 28/05/2008 FATE - Marco Pennacchiotti 10 / 17

Annotation Schema Span Annotation • On T of positive pairs, annotate only the fragments (spans) contributing to the inferential process – Spans are obtained from the ARTE annotation [Garoufi, 2007] – For negative pairs it is not straightforward to derive spans, hence we do full annotation T: “Soon after the EZLN had returned to Chiapas, Congress approved a different version of the COCOPA Law, which did not include the autonomy clauses, claiming they were in contradiction with some constitutional rights (private property and secret voting); this was seen as a betrayal by the EZLN and other political groups. ” H: “EZLN is a political group. ” 28/05/2008 FATE - Marco Pennacchiotti 11 / 17

Annotation Schema Other guidelines • Unknown frames: use an UNKNOWN frame for words evoking situations not present in the Frame. Net database • Anaphora • Copula and support verbs • Modal expressions • Metaphors • Existential constructions • … 28/05/2008 FATE - Marco Pennacchiotti 12 / 17

Corpus statistics • Annotated pairs : 800 (400 positive, 400 negatives) • Annotated frames : 4, 500 : avg. 5. 6 frames per pair : 1, 600 frames in positive pairs : 2, 800 in negative pairs • Annotated roles : 9, 500 : avg. 2. 1 roles per frame • Annotation time : 230 hours : 90 h for positive pairs (13 min/pair) : 140 h for negative pairs (21 min/pair) 28/05/2008 FATE - Marco Pennacchiotti 13 / 17

Frame. Net and RTE (simple case) T H • Syntactic normalization – Active / Passive EDUCATIONAL_TEACHING(STUDENT: ground soldiers / soldiers; MATERIAL: virtual reality/ virtual reality) 28/05/2008 FATE - Marco Pennacchiotti 14 / 17

Implementation gap insights (1) Resource coverage is too low (2) Models for predicate-argument inference are weak (3) Automatic annotation models (SRL) are not good enough to be safely used in RTE • Frame. Net coverage is good: – 373 Unknown frames (8 % of total frames) – Unknown roles 1 % of total roles • Coverage is unlikely to be a limiting factor for using Frame. Net in applications 28/05/2008 FATE - Marco Pennacchiotti 15 / 17

Why should you use FATE ? (1) Resource coverage is too low (2) Models for predicate-argument inference are weak (3) Automatic annotation models (SRL) are not good enough to be safely used in RTE • To better study predicate-argument inference in RTE • To experiment frame-RTE models on a gold-std corpus • To learn better SRL models, by training on FATE Corpus is freely available on-line 28/05/2008 FATE - Marco Pennacchiotti 16 / 17

Thank you! Questions? FATE download: http: //www. coli. uni-saarland. de/projects/salsa/fate pennacchiotti@coli. unisb. de 28/03/2008 FATE – Marco Pennacchiotti www. coli. uni- 17 / 17