METIS Traducci Automtica per a llenges amb pocs

METIS (Traducció Automàtica per a llengües amb pocs recursos ) Maite Melero (GLi. Com – BM) 23/05/2008 Seminari NLP-UPC 1

Roadmap METIS II (2004 -2007) ES-EN approach (GLi. Com) METIS II evaluation results Rapid deployment of METIS CA-EN pair 23/05/2008 Seminari NLP-UPC 2

Current approaches to MT In industry: mainly rule-based è require lots of expensive manual labour In academia: mostly data driven (statistical and example-based MT) è require large parallel corpora What happens with smaller languages? 23/05/2008 Seminari NLP-UPC 3

METIS II (2004 -2007): the aims Construct free text translations by n n n relying on hybrid techniques employing basic resources retrieving the basic stock for translations from large monolingual corpora of the target language only 23/05/2008 Seminari NLP-UPC 4

Similar approach: MATADOR (Habash and Dorr, 2002, 2003; Habash, 2003, 2004). Main difference: n n MATADOR aims at language pairs with resource asymmetry: low resources for the source language, and high resources for the target language METIS aims at low resources on both sides 23/05/2008 Seminari NLP-UPC 5

Metis II: The main ideas Hybrid approach: strong data driven component plus a limited number of rules Simple resources, readily available Weights associated with resources and the search algorithm TL corpus: processed off-line to construct TL model Language-specific components independent from the core search engine Special data format for the core engine input (UDF) Several language pairs test feasibility of the approach: Dutch, German, Greek and Spanish English. 23/05/2008 Seminari NLP-UPC 6

METIS II architecture 23/05/2008 Seminari NLP-UPC 7

What are basic NLP resources? Part-of-speech taggers Lemmatizers Manually corrected POS tagged corpus (can be used to train a statistical tagger such as Tn. T (Brants, 2000)) (optionally) Chunkers 23/05/2008 Seminari NLP-UPC 8

Metis II: Fields of experimentation • • • SL analysis w depth & richness of syntactic structure Transfer w which pieces / structures of information Generation w re-ordering of chunks and words 23/05/2008 Seminari NLP-UPC 9

Metis II: SL Analysis (Morphology) All language pairs provide: n Lemmatisation: w abstraction from inflection n POS tagging: w verb, noun, adjectives, articles, pronouns, etc, with subclasses according to properties of SL n Nominal Inflection: w number, gender, case n Verbal Inflection: w number, person, tense, mood, type (ptc, fin, inf, etc. ) 23/05/2008 Seminari NLP-UPC 10

Metis II: SL Analysis (Syntax) No syntactic SL analysis: Spanish Phrase detection (nominal, prepositional, verbal groups) and Clause detection (main and subordinate clause): Dutch, German & Greek Recursive embedding of phrases and clauses: n one level, no embedding: German n two level embedding: Greek n full recursivity: Dutch detect phrase & clause head: Dutch & Greek subject detection: German & Greek topological field analysis: German 23/05/2008 Seminari NLP-UPC 11

Metis II: Source Language Analysis Provides generalization: • Smaller lexicon • Less data sparsity in TL corpus 23/05/2008 Seminari NLP-UPC 12

Metis II: Transfer (Mapping of SL features to TL) 23/05/2008 Seminari NLP-UPC 13

Metis II: TL Generation (Reordering) Reordering of the transferred items into TL structure is conceived as a process of hypothesis generation and filtering, according to most likely TL pattern (from TL model). Mostly pattern-based and use only info from TL, but n can also be partly rule-based and use information from SL (Dutch and German) 23/05/2008 Seminari NLP-UPC 14

Metis II: TL Generation (Reordering) Information to be matched in TL model w Shallow syntactic information: all exc. Spanish w n-gram patterns of mapped Pos & lemmata: Spanish Matching Procedure w top down: Greek w bottom up: all exc. Greek 23/05/2008 Seminari NLP-UPC 15

Metis II: Reordering mechanism for TL word order generation 23/05/2008 Seminari NLP-UPC 16

Metis II Spanish-English Translation Paradigm Spanish sentence Spanish Preprocessing POS tagger and lemmatizer Translation Model Bilingual flat lexicon (no structure transfer rules) English Generation Search over ngram models extracted from English corpus English sentence 23/05/2008 Seminari NLP-UPC 17

Main Translation Problems Lexical selection: i. e. picking the right translation for a given word n n escribir una carta write a letter jugar una carta play a card Translation divergences: i. e. whenever wordby-word translation does not work n n ver a Juan see (to) Juan cruzar nadando cross swimming (swim across) 23/05/2008 Seminari NLP-UPC 18

Translation Divergences. How MT has addressed them Linguistic based MT systems devise data representations that minimize translation divergences. [head] ver [arg 2] Juan n [head] see [arg 2] Juan Remaining divergences need to be solved in the translation module: n n Hand written bilingual mapping rules (Transfer MT). Mappings automatically extracted from parallel corpus (Example Based MT). 23/05/2008 Seminari NLP-UPC 19

Translation Divergences. Our constraints. Very basic resources required, both for source and target languages: only lemmatizer -POS tagger and (TL) chunker. No deep linguistic analysis to minimize divergences è No parallel corpus, only target corpus è Keep translation model very simple: only bilingual lexicon. è No mapping rules, either hand-written, or automatically learned. 23/05/2008 Seminari NLP-UPC 20

Translation Divergences. Our approach. Handle structure modifications in the TL Generation component. Treatment independent of the SL, i. e. much more general and reusable. 23/05/2008 Seminari NLP-UPC 21

SL Preprocessing (Spanish) Tagger (Cast. CG) Statistical disambiguation SL normalization 23/05/2008 Seminari NLP-UPC 22

Spanish Tagger: Cast. CG Me alojo en la casa de huéspedes. Form Lemma Synt. rels Synt. tags Morph tags Me me obj>2 @NH PRON Pers SG P 1 ACC alojo alojar main>0 @MAIN V IND PRES SG P 1 en en pm>5 @PREMARK PREP la la det>5 @PREMOD DET FEM SG casa loc>2 @NH N FEM SG de de pm>7 @POSTMOD PREP huéspedes huésped mod>5 @NH N MSC PL 23/05/2008 Seminari NLP-UPC 23

SL Normalization: Tag Mapping Form Lemma POS (PAROLE) Morph tag Me me PP sg: 1: acc alojo alojar VM i: p: sg: 1 en en SP la la TD f: sg casa NCF f: sg de de SP huéspedes huésped NCC 23/05/2008 Seminari NLP-UPC m: pl 24

SL Normalization: e. g. Pronoun Insertion in Pro-drop Form Lemma POS (PAROLE) Morph tag yo yo PP sg: 1: nom Me me PP sg: 1: acc alojo alojar VM i: p: sg: 1 en en SP la la TD f: sg casa NCF f: sg de de SP huéspedes huésped NCC 23/05/2008 Seminari NLP-UPC m: pl 25

Translation Model: Spanish. English Lexicon Look-up Oxford Spanish lemma Sp. POS (PAROLE) English lemma Eng. POS (reduced CLAWS 5) Order of translation alojar VM house VV 1 alojar VM stay VV 2 alojar 23/05/2008 VM lodge VV Seminari NLP-UPC 3 Sp-Eng lexmetis HD HD List of Pseudo. English candidates (UDF) 26

Translation Model: Compound Detection <trans-unit id="6"> <option id="1"> <token-trans id="1"> Oxford casa => house <lemma>boarding</lemma> <pos>VVG</pos> </token-trans> casa => casa de huéspedes <token-trans id="2"> Sp-Eng lexmetis <lemma>house</lemma> <pos>NN 1</pos> </token-trans> </option> casa de huéspedes => boarding house </trans-unit> 23/05/2008 Seminari NLP-UPC 27

Translation Model: Unfound words Past participle Ex. “denominado” > denominar (VM) > designate (VV) > designated (AJ 0) Adverbs Ex. “técnicamente” > técnico (AQ) > technical (AJ 0) > technically (AV 0) 23/05/2008 Seminari NLP-UPC 28

TL Generation (English) Pseudo-English UDF Search Engine (TL models) English lemmatized sentence Token generation English translation 23/05/2008 Seminari NLP-UPC 29

Search Engine (1 st version) Lexical preselection Candidate expansion Candidate scoring 23/05/2008 Seminari NLP-UPC n-gram TL models 30

Search Engine (2 nd version): beam search decoding the worker must Search engine carry helmet wear bottle drive headphones … helmet Lexical pre-selection Candidate expansion Scoring n-gram TL models 23/05/2008 Seminari NLP-UPC 31

Handling Structure Divergences in TL Generation: Local Structure Modifications n freq I 1 -gram 2 -gram 3 -gram 4 -gram 5 -gram D n freq want|VV go|VV want|VV to|TO 0 go|VV at|PRP the|AT 0 home|NN at|PRP home|NN • Insertion of functional words: want|VV to|TO 0 go|VV • Deletion of functional words: at|PRP (the|AT 0) home|NN • Permutation of content words: a|AT 0 {day|NN happy|AJ 0} 23/05/2008 Seminari NLP-UPC 33

Search Engine: beam search decoding Performance problems § Combinatorial explosion in the expansion step: Suppose we are given a source sentence with at least 35 words which translate to at least to English words. Thus: 23/05/2008 Seminari NLP-UPC 34

Search Engine: beam search decoding Performance problems § Combinatorial explosion in the expansion step: Suppose we are given a source sentence with at least 35 words which translate to at least to English words. Thus: The search space of candidates must be pruned. 23/05/2008 Seminari NLP-UPC 35

Search Engine: beam search decoding Performance problems § Combinatorial explosion in the expansion step § Combinatorial explosion in the scoring computation step. 23/05/2008 Seminari NLP-UPC 36

Search Engine: beam search decoding Solution: § To incrementally build the search space (following Philipp Koehn’s Pharaoh: a Beam Search Decoder for Phrase-Based Statistical Machine Translation Models) (2004) 23/05/2008 Seminari NLP-UPC 37

Search Engine: beam search decoding § To incrementally build the search space (following Philipp Koehn’s Pharaoh: a Beam Search Decoder for Phrase-Based Statistical Machine Translation Models) (2004) w 1, …, wk are pushed on the first stack. The stack is ranked and pruned up to a given stack depth Solution: 23/05/2008 Seminari NLP-UPC 38

Search Engine: beam search decoding § To incrementally build the search space (following Philipp Koehn’s Pharaoh: a Beam Search Decoder for Phrase-Based Statistical Machine Translation Models) (2004) Each candidate of (i-1)-th stack is expanded via the dictionary and edit ops. Again Candidates are ranked and pruned up to given stack depth. Solution: 23/05/2008 Seminari NLP-UPC 39

Search Engine: beam search decoding § To incrementally build the search space (following Philipp Koehn’s Pharaoh: a Beam Search Decoder for Phrase-Based Statistical Machine Translation Models) (2004) The scoring of each partial translations is computed using the already computed stored scorings: Solution: 23/05/2008 Seminari NLP-UPC 40

Search Engine: beam search decoding § To incrementally build the search space (following Philipp Koehn’s Pharaoh: a Beam Search Decoder for Phrase-Based Statistical Machine Translation Models) (2004) § At the N-th step (the source sentence contains N tokens) the decoding process stops. We get a ranked stack with the translation candidates. Solution: 23/05/2008 Seminari NLP-UPC 41

Handling Structure Divergences in TL Generation: Non-local Movements Normalized BNC Syntactic model Boundaries PRPAT 0 NN VV AT 0 NN PRPAT 0 NN VV PRPAT 0 NN Chunked Corpus 23/05/2008 Seminari NLP-UPC eg. [The man] [sleeps] [at the park] 42

Evaluation final METIS prototype Comparison with SYSTRAN: n n n Widely used Available for all language pairs Rule-based, many man-years of development Goal: get an estimation of what has been achieved 23/05/2008 Seminari NLP-UPC 43

Methodology: Test sets Two test sets: n n 200 sentences manually chosen from Europarl 200 sentences from balanced test suite used to validate system development from a variety of domains: 23/05/2008 w 25% grammatical phenomena newspapers technical scientific Seminari NLP-UPC 44

Methodology: Metrics BLEU & NIST: measure edit distance using Ngrams TER (Translation Error Rate): measures the amount of editing that a human would have to perform 23/05/2008 Seminari NLP-UPC 45

Methodology: References All metrics use human created references to compare with MT output. Europarl: 5 references (4 resulting from human translating each SL into English + original English one) Development: 3 references 23/05/2008 Seminari NLP-UPC 46

Results on Europarl test set METIS-II SYSTRAN difference % NL-EN 0. 1925 0. 3828 0. 1903 50% DE-EN 0. 2816 0. 3958 0. 1142 71% EL-EN 0. 1861 0. 3132 0. 1271 59% ES-EN 0. 2784 0. 4638 0. 1854 60% BLEU 23/05/2008 Seminari NLP-UPC 47

Results on development test suite NL-EN DE-EN EL-EN ES-EN METIS-II SYSTRAN difference % 0. 2369 0. 3777 0. 1408 70% 0. 2231 0. 3133 0. 0902 71% 0. 3661 0. 3946 0. 0285 92% 0. 2941 0. 4634 0. 1693 63% BLEU 23/05/2008 Seminari NLP-UPC 48

METIS-II on both test sets NL-EN DE-EN EL-EN Europarl 0. 1925 0. 2816 0. 1861 ES-EN 0. 2784 Dev difference 0. 2369 0. 0444 0. 2231 -0. 0585 0. 3661 0. 1800 0. 2941 0. 0157 BLEU 23/05/2008 Seminari NLP-UPC 49

Results according to text type (ES-EN on Development Testsuite) Grammar News Science Tech METIS-II 0. 22 0. 33 0. 29 0. 26 SYSTRAN 0. 48 0. 46 0. 47 0. 45 BLEU 23/05/2008 Seminari NLP-UPC 50

Impact of the number of reference translations (DE-EN on Europarl) Ref 1 (Greek) 1 (Dutch) 1 (Europarl) 1 (Spanish) 1 (German) 4 (- Spanish) 4 (- Dutch) 4 (- Greek) All 5 23/05/2008 METIS-II 0. 0761 0. 0923 0. 1155 0. 1199 0. 2376 0. 1803 0. 2697 0. 2750 0. 2774 0. 2816 Seminari NLP-UPC SYSTRAN 0. 1182 0. 1483 0. 1975 0. 1922 0. 2912 0. 3064 0. 3739 0. 3817 0. 3871 0. 3958 51

Conclusions Homogeneity of results confirm language independence of the strategy Young Metis II still not at the same level as mature SYSTRAN, but … … stands to the comparison. Ample room for improvement 23/05/2008 Seminari NLP-UPC 52

Lines for (quick) improvement Fix bugs Extend lexical coverage Augment Target corpus Add mapping rules 23/05/2008 Seminari NLP-UPC 53

Other lines for improvement (need bilingual corpora…) Parameter tuning Enrichment of the translation model n n Automatic induction of dictionary entries (lexical and phrasal) Automatic induction of structure transfer rules Automatic postediting rules (needs corpus of corrected translations) 23/05/2008 Seminari NLP-UPC 54

Publications (METIS-II) (1) Toni Badia, Gemma Boleda, Maite Melero, and Antoni Oliver, An ngram approach to exploiting monolingual corpus for MT, in: Proceedings of the 2 nd Workshop on Example-Based Machine Translation held in conjunction with the 10 th Machine Translation Summit, pp. 1 -8, Phuket, 2005. Toni Badia, Gemma Boleda, Maite Melero, and Antoni Oliver, El proyecto METIS-II, in: Procesamiento del Lenguaje Natural, 35 (2005), pp. 443 -444. Antonio Oliver, Toni Badia, Gemma Boleda, and Maite Melero, Traducción automática estadística basada en n-gramas, in: Procesamiento del Lenguaje Natural, 35 (2005), pp. 77 -84. Vincent Vandeghinste, Ineke Schuurman, Michael Carl, Stella Markantonatou, and Toni Badia, METIS-II: Machine Translation for Low-Resource Languages, in: Proceedings of the Fifth International Conference on Language Resources and Evaluation, pp. 1284 -1289, Genoa, 2006. 23/05/2008 Seminari NLP-UPC 55

Publications (METIS-II) (2) Maite Melero, Antoni Oliver, Toni Badia, and Teresa Suñol, Dealing with Bilingual Divergences in MT using Target Language N-gram Models, in: Proceedings of the METIS-II Workshop: New Approaches to Machine Translation, pp. 19 -26, Leuven, 2007. Maite Melero and Toni Badia, Demonstration of the Spanish to English METIS-II MT System, in: Proceedings of the 11 th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI), pp. 132 -133, Skövde, 2007. Toni Badia, Maite Melero and Oriol Valentin, Rapid deployment of a new METIS language pair: Catalan-English. Submitted to LREC, Marrakech, 2008. 23/05/2008 Seminari NLP-UPC 56

Follow-Up Projects AMASS++: Summarization of multimedia / multilingual information (KUL): NL <-> EN Pa. Co-MT: Parse and Corpus based MT (KUL): NL <-> EN NL <-> FR I 3 Media (FUPF) ES -> CA CA -> EN EN -> ES 23/05/2008 Seminari NLP-UPC 57

Rapid deployment of new METIS language pair: CA-EN Catalan-English prototype Motivation: Parallel corpora even harder to get for smaller languages 23/05/2008 Seminari NLP-UPC 58

Rapid deployment of new METIS language pair: CA-EN We have simply plugged to the English Generation: n n A Catalan POS tagger (Cat. CG) An open source CA-EN dictionary (Dacco) Adaptation and integration: less than 1 person / month 23/05/2008 Seminari NLP-UPC 59

Experiment and evaluation Translated to Catalan 200 Spanish sentences from METIS II development testsuite Grammar News Science Tech All Cat-Eng METIS 0. 2059 0. 2533 0. 2070 0. 2365 0. 2342 Sp-Eng METIS 0. 2241 0. 3273 0. 2876 0. 2633 0. 2941 Cat-Eng Translendium 0. 3334 0. 4406 0. 4226 0. 4264 0. 4250 23/05/2008 Seminari NLP-UPC 60

Gràcies! 23/05/2008 Seminari NLP-UPC 61