The Effect of Pseudo Relevance Feedback on MTBased
The Effect of Pseudo Relevance Feedback on MT-Based CLIR Yan Qu, Alla N. Eilerman Hongming Jin, David A. Evans CLARITECH Corporation The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000
Outline • Our approach to Cross-Language Information Retrieval (CLIR) • Objectives of this work • Review of previous work with Pseudo Relevance Feedback (PRF) • System diagram • Data for experiments • Error analysis of MT-based query translation • The effect of PRF on French monolingual retrieval • The effect of PRF on English-to-French crosslanguage retrieval • Summary and conclusions The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 2
Our Approach to CLIR • Used MT-based query translation to bridge the language gap • Adapted pseudo relevance feedback to CLIR – pre-translation query expansion – post-translation query expansion – combined (pre- and post-translation) query expansion The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 3
Objectives • Identify factors that affect the quality of MT-based query translation • Evaluate the effectiveness of using pseudo relevance feedback for improving CLIR performance • Identify contexts for selecting these feedback methods The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 4
Relevance Feedback in Monolingual Retrieval • Relevance feedback (Salton & Buckley, 1990; Evans et al. , 1999) • Pseudo relevance feedback (PRF) (Evans & Lefferts, 1994; Milic-Frayling et al. , 1998) • Both have been demonstrated to be effective in improving retrieval performance The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 5
Pseudo Relevance Feedback in CLIR The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 7
CLIR with Simple MT-based Query Translation Queries in SL MT Queries in TL Retrieval Ranked list from TL Database The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 8
CLIR with Query Expansion Before MT Queries in SL Retrieval & Thesaurus Extraction MT Queries in TL & thesaurus terms in TL Thesaurus Terms in SL Retrieval Ranked list from TL Database The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 9
CLIR with Query Expansion After MT Queries in SL MT Queries in TL Retrieval & Thesaurus Extraction Retrieval Ranked list from TL Database Thesaurus Terms in TL The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 10
Process Summary The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 11
CLARIT English NLP • Used for processing the English corpus and the English queries • Consists of a parser and morphological analyzer • Uses an English lexicon and grammar to identify linguistic structures in texts • Supplemented by a “stop word” list to filter out substantive words that are extraneous to the topics (e. g. , document, relevant) The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 13
French Text Processing (Pseudo-NLP Approach) • Goal: to obtain mostly correct phrase segmentation • Manually constructed resources – lexicon of closed-class categories with 1081 entries – “stop word” lexicon including 525 words and their inflected forms that are extraneous to the topics (e. g. , document, pertinent) – grammar based on the CLARIT English grammar and adapted to accommodate French categories – no French morphological normalization The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 14
English-to-French Translation • SYSTRAN Enterprise translation software • Translation direction: English to French • Client-server architecture • Translation is a black box to our system • No special or additional resources were used to supplement the translation process The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 15
Data Sources for Experiments • TREC-6 CLIR track data collections provided by NIST (Voorhees & Harman, 1998) – 250 MB collection of French SDA news (1988 -1990) from the Swiss News Agency: 141, 656 documents – 750 MB collection of English AP news (1988 -1990) from the Associated Press: 242, 918 documents The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 16
Topics for Experiments • TREC-6 CLIR track topics provided by NIST (Voorhees & Harman, 1998) – 22 English topics for the English-to-French cross-language runs – 22 French topics for the French monolingual runs – Equivalent across languages – Prepared by humans – Composed of the title, description, and the narrative fields The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 17
A Sample English Topic <num> Number: CL 1 <E-title> Waldheim Affair <E-desc> Description: Reasons for controversy surrounding Waldheim's World War II actions. <E-narr> Narrative: Revelations about Austrian President Kurt Waldheim’s participation in Nazi crimes during World War II are argued on both sides. Relevant documents are those that express doubts about the truth of these revelations. Documents that just discuss the affair are not relevant. The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 18
An Ideal French Topic <num> Number: CL 1 <F-title> Affaire Waldheim <F-desc> Description: Raisons de la controverse à l'égard des agissements de Waldheim pendant la deuxième guerre mondiale. <F-narr> Narrative: Les révélations sur la participation du président autrichien Kurt Waldheim aux crimes nazis pendant la deuxième guerre mondiale font l'objet de controverses. Les documents pertinents font état de doutes sur la culpabilité de Waldheim. Les articles qui ne font que mentionner l'affaire ne sont pas valables. The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 19
CLARIT Queries • Composed of the title, description, and the narrative fields • Processed automatically into query vectors The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 20
Sample English Query Vector <cf="1" tf="1">waldheim affair</> <cf="1" tf="1">waldheim world war ii</> <cf="1" tf="1">nazi crime</> <cf="1" tf="1">austrian president kurt waldheim</> <cf="1" tf="1">austrian president</> <cf="1" tf="1">controversy surround</> <cf="1" tf="1">president kurt waldheim</> <cf="1" tf="1">kurt waldheim</> <cf="1" tf="3">waldheim</> <cf="1" tf="1">kurt</> <cf="1" tf="2">revelation</> <cf="1" tf="1">austrian</> <cf="1" tf="1">participation</> <cf="1" tf="1">surround</> <cf="1" tf="1">truth</> The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 21
Sample French Query Vector <cf="1" tf="1">crimes nazis</> <cf="1" tf="1">affaire waldheim</> <cf="1" tf="1">président autrichien kurt waldheim</> <cf="1" tf="1">président autrichien</> <cf="1" tf="1">controverses</> <cf="1" tf="1">agissements</> <cf="1" tf="1">kurt waldheim</> <cf="1" tf="1">culpabilité</> <cf="1" tf="4">waldheim</> <cf="1" tf=” 2">deuxième guerre mondiale</> <cf="1" tf="2">deuxième guerre</> <cf="1" tf="1">doutes</> <cf="1" tf="1">révélations</> <cf="1" tf="1">nazis</> The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 22
Topic and Query Statistics The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 23
Evaluation • Relevance judgments on the French SDA news, prepared by NIST judges (TREC-6) • Evaluation measures: – eleven-point average precision (N=1000 documents) – precision at low recall levels (10, 20, and 100 documents) – recall – exact precision The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 24
English-to-French Retrieval vs. French Monolingual Retrieval (without PRF) The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 25
Types of Translation Errors • E 1: missing translation of an English term • E 2: unnecessary translation of a borrowed English term • E 3: wrong sense disambiguation • E 4: wrong sense disambiguation caused by removed capitalization • E 5: word-by-word translation of a multiword (idiomatic) term • E 6: wrong phrase construction • E 7: broken phrase The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 26
Error Type 1: Missing Translation English: agencies’ Ideal French translation: (des) agences MT output: (d’)agencies The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 27
Error Type 2: Unnecessary Translation English: fast food Ideal French translation: fast food MT output: aliments de préparation rapide (food of fast preparation) The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 28
Error Type 3: Wrong Sense Disambiguation English: logging Ideal French translation: déforestation (deforestation) MT output: notation (notation) The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 29
Error Type 4: Wrong Disambiguation Caused by Removed Capitalization English: aids (AIDS) Ideal French translation: sida (SIDA “AIDS”) MT output: aides (assistants) The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 30
Error Type 5: Word-by-Word Translation of a Multiword Idiomatic Term English: death penalty Ideal French translation: la peine de mort MT output: la pénalité de la mort The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 31
Error Type 6: Wrong Phrase Construction English: austrian president kurt waldheim’s participation Ideal French translation: la participation du président autrichien kurt waldheim MT output: la participation autrichienne de waldheim de kurt de président The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 32
Error Type 7: Broken Phrase English: sex education Ideal French translation: éducation sexuelle MT output: éducation de sexe The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 33
Error Distributions Wrong sense disambiguation Frequency 25 20 15 Word-by-word translation Wrong phrase construction Broken phrases 10 5 Frequency 0 E 1 E 2 E 3 E 4 E 5 E 6 E 7 Error Type The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 34
The Effect of PRF on French Monolingual Retrieval The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 35
The Effect of PRF on English-to-French Retrieval The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 36
English-to-French Retrieval vs French Monolingual Retrieval (with PRF) The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 37
Cross-Language Retrieval vs. Monolingual Retrieval The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 38
Cross-Language Retrieval vs. Monolingual Retrieval The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 39
Performance of Different PRF Methods Topic 1009: Effects of logging Topic 1016: Tuberculosis The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 40
Topic 1009 “Effects of Logging” • Key concept lost due to wrong sense disambiguation (E 3 error): logging (felling trees) notation (notation) • Pre-translation feedback – neutralized the effect of the translation error by bringing useful thesaurus terms (tropical forest, tree, earth, sea, ocean, land, atmosphere, carbone dioxide, ozone depletion, greenhouse effect, global warming, destruction, pollution, damage, environmentalist, conference, organization, world, nation, country). Result: 688% increase in average precision • Post-translation feedback – returned some useful terms – introduced noise caused by the wrong translation of logging Result: 29% increase in average precision • Combined feedback – created a strong base query prior to translation – further improved it with appropriate terms after translation – avoided too much noise Result: 621% increase in average precision The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 41
Topic 1016 “Tuberculosis” • Key term is translated correctly: tuberculosis tuberculose • Translation errors affected some important terms: aids (AIDS) aides (assistants) third-world (countries) le troisième-monde (the third world) • Pre-translation and combined feedback created additional sources of errors and noise by introducing – ambiguous thesaurus terms (cases, tests), which were mistranslated (caisse instead of cas, essai instead of test) – acronyms (AIDS, CDC, HIV), either mistranslated or not translated Result: 29 -30% decrease in average precision • Post-translation feedback compensated for translation errors by bringing – correct terms (SIDA “AIDS”, tiers monde “third world) – additional useful terms (bacille, tuberculeux, virus, infectées, maladie, risque, santé, problème, etc. ) Result: 32% increase in average precision The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 44
Performance of Different PRF Methods The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 45
Decision Tree for Selecting PRF Methods Most key terms translated correctly Yes All three methods behave similarly; generally improve retrieval performance No Lost of meaning compensated by context terms Yes Post-MT feedback is generally better than pre-MT and combined feedback No Pre-MT and combined feedback are generally better than post-MT feedback The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 48
Summary • Adopted pseudo relevance feedback for query expansion in CLIR with MT-based query translation • Conducted analysis of translation errors • Evaluated empirically the effect of three feedback methods on retrieval performance • Examined contexts where different feedback methods are effective The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 49
Conclusions • Wrong sense disambiguation and inappropriate translation of multi-word terms are the most frequent translation errors when using MT. • All feedback methods demonstrated significant performance improvement in CLIR compared with not using feedback. • The use of PRF in general helps to reduce the negative effect of translation errors. • Post-translation feedback generally outperforms pretranslation and combined feedback. • The effectiveness of different feedback methods depends on the types of translation errors and the relative importance of the terms affected by these errors. The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 50
Future Work • Investigate the effect of query length • Investigate the effect of context • Develop measures to evaluate the original query quality • Develop measures to evaluate the translated query quality • Investigate the empirical conditions for selecting different feedback methods The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 51
The End The Effect of Pseudo Relevance Feedback on MT-Based CLIR © 2000, CLARITECH Corporation April 12, 2000 52
- Slides: 46