Joint Sem EvalCLEF tasks Contribution of WSD to

Joint Sem. Eval/CLEF tasks: Contribution of WSD to CLIR UBC: Agirre, Lopez de Lacalle, Otegi, Rigau, FBK: Magnini Irion Technologies: Vossen CLEF 2007 - Budapest

WSD and Sem. Eval n Word Sense Disambiguation n When I went to bed at around two o'clock that night , everyone else was still out in the party. n party: N: 1 political organization n party: N: 2 social event n Potential for more precise expansion (translation) n Sem. Eval 2007 n Framework for semantic evaluations n Under auspices of SIGLEX (ACL) n 19 tasks incl. WSD, SRL, full frames, people, … n > 100 attendants in ACL workshop CLEF 2007 - Budapest 2

Motivation for the task n WSD perspective In-vitro evaluations not fully satisfactory n In-vivo evaluations in applications (MT, IR, …) n IR perspective n Usefulness of WSD on IR/CLIR disputed, but … n Real compared to artificial experiments n Expansion compared to just WSD n Weighted list of senses compared to best sense n Controlling which word to disambiguate n WSD technology has improved n Coarser-grained senses (90% acc. on Semeval 2007) n CLEF 2007 - Budapest 3

Motivation for the task n Combining WSD and IR: n Many possible variations n Unfeasible for a single research team n A public common dataset allows for the community to explore different combinations. n Tasks where we could hope to get positive impact: n High recall IR scenarios n Short passage IR scenarios n Q&A n CLIR n We selected CLIR because of previous expertise of some of the organizers. CLEF 2007 - Budapest 4

Two-stage framework n First stage (Sem. Eval 2007 task 01): n Participants: submit WSD results n Sense inventory Word. Net 1. 6 (multilinguality) n Organizers: n Expansion / translation strategy fixed n IR/CLIR system fixed (IR as upperbound) n Second stage (Proposed CLEF 2008 track): n Organizers: provide several WSD annotations n Participants: submit CLIR results with/without WSD annotations CLEF 2007 - Budapest 5

Outline n Description of the Sem. Eval task (1 st stage) n Evaluation of results (1 st stage) n Conclusions (1 st stage) n Next step (2 nd stage) CLEF 2007 - Budapest 6

Description of the task Datasets n CLEF data: n Documents in English: LA 94, GH 95 170. 000 documents, 580 Mb raw text n 300 topics: both in English and Spanish n Existing relevance judgments n Due to time limitations of the exercise n 16, 6% of document collection (we will have 100% shortly) n Subset of relevance judgments, 201 topics CLEF 2007 - Budapest 7

Description of the task Two subtasks for participants English WSD of the following: n the document collection n the topics We limit to English at the time being. Return WN 1. 6 senses. CLEF 2007 - Budapest 8

Description of the task Steps of CLIR/IR system Step 1: Participants return WSD results Step 2: Expansion / Translation n Multilingual Central Repository (based on Euro. WN) 5 languages tightly connected n To ILI concepts (WN 1. 6 synsets) n Mappings to other WN versions n n Example: car sense 1 n Expanded to synonyms: automobile n Translated to equivalents: auto, coche CLEF 2007 - Budapest 9

Description of the task Steps of CLIR/IR system Step 3: IR/CLIR system n Adaptation of Twenty. One (Irion) Pre-processing: XML n Indexing: detected noun phrases only n Title and description used for queries n Stripped down to vector-space matching n CLEF 2007 - Budapest 10

Description of the task Three evaluation settings n IR with WSD of documents (English) n WSD of English documents n Expansion of senses in the documents n IR with WSD of topics (English) n WSD of English documents n Expansion of senses in the documents IR as upperbound of CLIR n CLIR with WSD of documents: n n n WSD of English documents Translation of English documents Retrieval using Spanish topics n CLIR with WSD of topics (Spanish WSD, NO) CLEF 2007 - Budapest 11

Evaluation and results Participant systems n Participants returned sense-tagged documents and topics n Two systems participated: n n PUTOP from Princeton, unsupervised UNIBA from Bari, KB using Word. Net n In-house system: n ORGANIZERS, supervised, k. NN classifiers n Other baselines: n Noexp: original text n Fullexp: expand to all senses n WSDrand: return sense at random n 1 st: return first sense in Word. Net n Wsd 50: 50% best senses (in-house WSD system only) CLEF 2007 - Budapest 12

Evaluation and results S 2 AW and S 3 AW control Prec. Recall Senseval-2 all words ORG 0. 584 0. 577 UNIBA 0. 498 0. 375 PUTOP 0. 388 0. 240 Senseval-3 all words ORG 0. 591 0. 566 UNIBA 0. 484 0. 338 PUTOP 0. 334 0. 186 Cov. 93. 61% 75. 39% 61. 92% 95. 76% 69. 98% 55. 68% n Indication of performance of WSD n Not necessarily correlated with IR/CLIR results CLEF 2007 - Budapest 13

Evaluation and results Results (Mean Average Precision MAP) IR: noexp best CLIR: n fullexp best n ORG close n far from IR IRtops IRdocs CLIR noexp 0. 3599 0. 1446 fullexp 0. 1610 0. 1410 0. 2676 UNIBA 0. 3030 0. 1521 0. 1373 PUTOP 0. 3036 0. 1482 0. 1734 Wsdrand 0. 2673 0. 1482 0. 2617 1 st 0. 2862 0. 1172 0. 2637 ORG 0. 2886 0. 1587 0. 2664 wsd 50 0. 2651 0. 1479 0. 2640 Mean Average Precision Expansion and IR/CLIR system too simple CLEF 2007 - Budapest 14

Analysis # words in expansion of docs. IR: the less the better (but) n MAP: noexp > ORG > UNIBA n MW: noexp < … < ORG CLIR: the more the better (but) n MAP: fullexp > ORG (50) n MW: fullexp > ORG(50) > ORG WSD allows for more informed expansion Eng. noexp 9 NO WSD fullexp 93 wsdbest 19 UNIBA wsd 50 19 wsdbest 20 PUTOP wsd 50 20 1 st 24 Baseline wsdrand 24 ORG. wsdbest 26 wsd 50 36 Millions of words CLEF 2007 - Budapest Sp. 9 58 17 17 16 16 20 19 21 27 15

Conclusions n Main goals met: n First try on evaluating WSD on CLIR n Large dataset prepared and preprocessed n WSD allows for more informed expansion n On the negative side: n Participation low n Sem. Eval overload, 10 interested n No improvement over baseline n Expansion and IR/CLIR naive CLEF 2007 - Budapest 16

Next stage: CLEF 2008 n WSD results provided: n WSD of whole collection n Best WSD systems in Sem. Eval 2007 n CLEF teams will be able to try more sophisticated IR/CLIR methods n Feasibility of a Q/A exercise n Suggestions for cooperation on other tasks welcome Thank you for your attention! http: //ixa 2. si. ehu. es/semeval-clir CLEF 2007 - Budapest 17