Machine Translation as a Tool Prof dr Frieda

Machine Translation as a Tool Prof. dr. Frieda Steurs KU Leuven, Faculty of Arts

Agenda § Introduction § Predictions § MT usage & the market impact Translators’ profession § Quality § Productivity § Compensation § TM/MT integration § Small-scale experiment at KU Leuven § Portsmouth Conference 2014

About the presenter § § Postgraduate EMT in Specialised Translation Research Assistant for the SCATE project at KU Leuven Experience with training translators on CAT tools No special expertise in MT iulianna. vanderlekciudin@kuleuven. be Portsmouth Conference 2014

Smart computer-aided translation environment (SCATE) in a nutshell § § Time-frame: 01/03/2014 - 28/02/2018 Consortium: KU Leuven, 6 university teams, 10 companies Funding: Flemish government, industry, IWT Primary goal: Ø • Develop techniques to improve translation efficiency and consistency! How? Ø Better exploitation of resources, e. g. translation memories, terminology… § Syntax-based fuzzy matching § Automatic bilingual term extraction from comparable corpora Ø Stronger integration of speech recognition models with machine translation Ø Personalised user interfaces Portsmouth Conference 2014

SCATE Participants Project Consortium

SCATE Participants Industrial Advisory Committee

About Machine Translation 13 years ago…. A decent MT machine is just around the corner…A computer will never understand, but it can translate, at least to some extent. And, since translation without understanding is meaningless, the future of the human translator is proofsensing what a machine has pre-translated. Champollion (2001) In 6 years from now… We can expect that by 2020 automated translation tools will be embedded into every device, application and website. TAUS Portsmouth Conference 2014

Present Is the human translator proof-reading what a machine has pre-translated? YES Portsmouth Conference 2014

MT for Big Data Many languages Increasing content volumes Big Language Portsmouth Conference 2014

What about the LSP’s? …in the professional market, human translation still rules, though a quiet but pronounced shift to machine translation with post‐editing is unquestionably underway in the agencies, some of whom are also actively developing their own engines using open‐source software. LT-Innovate Report 2013 42% of the LSPs use MT Source: CSA Portsmouth Conference 2014

MT for Translation Rule-based Statistical • Linguistic rules • Specialised dictionaries per domain, customer, product line! • Suitable for software documentation, technical manuals, E-learning courseware, UI • Consistent and predictable quality • Out-of-domain translation quality • Lack of fluency • Hard to handle exceptions to rules • It does not cover all languages! • Expensive to develop and customize • Large bilingual corpora • Terminology choices hard to manage! • Suitable for user-generated content • Unpredictable translation quality • Poor out-of domain quality • Does not know grammar • High CPU and disk space required • Good fluency • Good for catching expeptions to rules • It covers all languages! • Rapid and cost-effective development Systran, Apertium, Babelfish, Prompt… Google, Moses, Language Weaver, Asia Online… Portsmouth Conference 2014 Hybrid • • Rules +Statistics Better output Solves ambiguity More expensive to manage two systems! Systran, Asia Online, Prompt

What is the market impact? In a nutshell: • • • Impact on the translation profession o Deprofessionalization of the translator? o New jobs, e. g. Post-Editor, MT quality evaluator… o How to maintain creativity? Quality o MT Process vs. quality of the MT output o What is the final expected quality? Productivity o • Raw MT output, Light post-editing vs full post-editing Compensation o Per word, per hour or both? Portsmouth Conference 2014

Benefits of MT for… End-clients o o Reduce costs Reduce time to market Increase productivity Translate new content Translators • • New opportunities Increase productivity Higher income More consistency Provided that: u. The MT process is implemented well u. Open dialogue between technology developers/Project Managers and translators/post-editors u. Clear guidelines for Quality, PE guidelines, PE rates… Portsmouth Conference 2014

What do translators say? …how to protect the language skills and mental function of those routinely exposed to toxic machine language. (Translator’s blog) I fear it might be the post-MT crash (of our industry, in terms of rates and quality). (Translator, Proz ) “post-processing of machine translations”, is likely to be much more deadly to the translating community, because this tool was designed by our beloved “translation industry leaders” to simply kill off the translating profession and replace it by a new type of workers called post-processors of machine translations. (Translator’s blog) Portsmouth Conference 2014

What do translators say? My proposal is that you start investing time and money to “update the skills” of your human base, current freelance translators and interpreters, so that you may “flip it” around. . I urge industry leaders to invite us, everyday translators and interpreters, to become an integral part (not a side story) of this equation. We have been severed from the most important conversations about our own future. Many of us are afraid of the new technologies because there is yet no clear answer to the question “what’s in it for me? ”. The “industry” as a whole may well benefit from inviting us to become part of the equation going forward. If translators and interpreters do not learn –quickly and swiftly– to use 21 st century technologies, we may not survive as a viable profession. Unless all the parties are on the same page, the “industry” as a whole may not evolve the way industry leaders might have anticipated. We need each other. Claudia Brauer (2014), blog post Will you help me flip it? Portsmouth Conference 2014

What do translators say? …it is safe to say that the number of professional translators who are using MT has risen in the past couple of years. I suspect that you would find large differences of usage among different language groups, type of projects, experience levels, and even in the way it is used. Significantly, MT seems to have transitioned into one of many productivity tools that are useful to some but not all, and I think the stigma it used to have among translators is gradually going away. ATA Chronicle (Feb. 2013) Conversation: Charlotte Brasler and Jost Zetzsche Portsmouth Conference 2014

What do you say? Portsmouth Conference 2014

Translation vs post-editing Translation process of a translator (TAUS 2010) Translation process of a post-editor (TAUS 2010) Portsmouth Conference 2014

Translators vs Post-editors Skills? Translators • • Post-editors • • Excellent word-processing and editing skills • • Knowledge of the SL, TL and the audience Specialised domain knowledge • Word-processing skills Ability to balance post-editing speed and effort with respect to the required quality level; • Knowledge of CAT environments and of the differences between post-editing MT output as opposed to editing the past human translations present in fuzzy matches. Fluent at least in two languages o Excellent knowledge of SL o Perfect command of TL Excellent written and verbal communication skills • • Understand the cultural differences • … Knowledge of language technologies Ability to work and make corrections directly on-screen Strong knowledge of general MT and its shortcomings Specific knowledge of the approach of the MT systems in use, especially their weaknesses and the reason for producing the encountered errors Ability to make quick decisions and reduce errors as needed Source: Doherty & Gaspari 2014, GALA blog Portsmouth Conference 2014

Quality Source: http: //www. multilingualtech. com/2014/05/08/language-translations-can-befunny-right/ Portsmouth Conference 2014

Quality is relative…The evaluation of the translation qualitywhether human or computerized – has to take into account the following intra-linguistic and inter-linguistic factors: morphology, syntax, content, terminology, style, conformity. (Bruderer 1978) Typically, there are many “perfect” translations of a given source sentence. These translations may vary in word choice or in word order even when they use the same words. And yet humans can clearly distinguish a good translations from a bad one. (Papineni et al. , 2002) Portsmouth Conference 2014

Quality • EN-15038: …simple to control! Lucija Biel, 2006 Portsmouth Conference 2014

Quality …more difficult to control! Source: Asia Online Portsmouth Conference 2014

Quality: MT Evaluation methods Human Automatic • No standardised method • Word Error Rate (WER) • Subjective Sentence Error Rate (SSER) • Metrics: o o o o Fluency and Adequacy (TAUS DQF, COSTA MTEvaluation, Appraise etc. ) o BLEU NIST WER METEOR LEPOR • Post-editing effort Portsmouth Conference 2014

Productivity Post-editing MT output can significantly increase the productivity of human translators! Guerberof 2009; Plitt and Masselot 2010; Federico et al. 2012; Green et al. 2013; Läubli et al. 2013 Portsmouth Conference 2014

Productivity TEP MT Post-editing • Translation: 250 words per • 350 -800 words per hour • … • Editing: 1000 words per hour • Proofing: 4000 words per hour Portsmouth Conference 2014

Productivity With Customised MT Source: Asia Online, Custom MT engines with Language Studio. TM Portsmouth Conference 2014

Productivity Autodesk 2010 Portsmouth Conference 2014

PE Compensation • Various experiments, no standard • Hourly rates or price per word? o o o 65%-85% of new word rate (Lionbridge) 40%-85% of new word rate (CSA report) $45/hr (Asia Online) Portsmouth Conference 2014

PE Compensation • It depends on certain factors: o o Quality of the MT output Translator’s productivity (speed) Language pair Content type Portsmouth Conference 2014

PE Compensation Less PE effort More words per hour Good MT output Portsmouth Conference 2014 More earnings?

TM/MT integration MT becomes even better when integrated in the translators TM environment! (O’Brien & Moorkens 2014) Ø A number of CAT tools connect to general and customized MT engines via API: Portsmouth Conference 2014

SDL Trados Studio 2014 memo. Q 2014 Portsmouth Conference 2014

Omega. T In the Omega. T user manual it is stated the following: If you do not wish to send your source segments to Google to get translated, untick the Google Translate menu entry in Options. Note that nothing but your source segment is sent to the MT service. The online version of Google Translate allows the user to correct the suggestion and send the corrected segment in. This feature, however, is not implemented in Omega. T. Portsmouth Conference 2014

Integrated PE analysis features Edit distance memo. Q 2014 Portsmouth Conference 2014

Post-edit Modifications Analysis in Trados Studio 2014 Portsmouth Conference 2014

Mate. Cat Portsmouth Conference 2014

Mem. Source Post-editing Analysis report Portsmouth Conference 2014

Post-editing analysis & Compensation Signs and Simptoms of Translation- Blog by Emma Goldsmith Portsmouth Conference 2014

Small-scale experiment at University of Leuven • Goal: o To test the TM/MT integration o To investigate the impact of MT on productivity and the final translation quality • Participants and language combinations: o 21 MA/Postgraduate students with EN > NL (Productivity test) o Translation Technology class Source text: o 5 short English texts, technical documentation, same level of difficulty Tools: o Trados Studio 2014 Resources: o TM with output from Google • • • Portsmouth Conference 2014

What did we measure? • Productivity (time): human translation vs post-editing MT • Quality: human-translations vs post-edited translations • The usefulness of MT • Translators’ expectations Portsmouth Conference 2014

How? 1. 2. 3. 4. 5. Translation from scratch, no TM/MT provided. Translation with TM/MT. Track the translation time of the whole project. Answer few questions about the usefulness of MT Portsmouth Conference 2014

Language combination used for the assignment: Portsmouth Conference 2014

Did you study English in the previous years? Portsmouth Conference 2014

Source text EN NL Portsmouth Conference 2014

Results Productivity Total average time per text for EN-NL Nr of words per text varied between 140200 Portsmouth Conference 2014

Results Quality Evaluation: Rate the translations on a scale from 1 -4 in terms of: o Clarity: How easily can you understand the translation? o Accuracy: How much of the meaning in the source text is also expressed in the translation? o Style: Is the language used in the translation appropriate for this content type? Fiederer & O’Brien (2009) Portsmouth Conference 2014

CLARITY: How easily can you understand the translation? In general Portsmouth Conference 2014

CLARITY: How easily can you understand the translation? Human Translation vs. Post-editing Portsmouth Conference 2014

ACCURACY: How much of the meaning in the source text is also expressed in the translation? In general Portsmouth Conference 2014

ACCURACY: How much of the meaning in the source text is also expressed in the translation? Human Translation vs. Post-editing Portsmouth Conference 2014

STYLE: Is the language used in the translation appropriate for a software product user manual? In general Portsmouth Conference 2014

STYLE: Is the language used in the translation appropriate for a software product user manual? Human Translation vs. Post-editing Portsmouth Conference 2014

General Usefulness of MT Final quality Terms Colloc ations Portsmouth Conference 2014

Did you find the machine translation output useful? Nr of students Portsmouth Conference 2014

Terminology? Collocations? 25 Nr of students 20 15 Very useless Useless 10 Useful Very useful 5 0 Useful for terminology collocations Portsmouth Conference 2014

Do you think the machine translation output had an impact on the quality of your final translation? Portsmouth Conference 2014

Is MT a disruptive technology? … or is it simply …. Portsmouth Conference 2014

What did students think of MT? Portsmouth Conference 2014

What did students think of MT? Portsmouth Conference 2014

Limitations • • Students had to record the time themselves Not all of them had a good knowledge of English Knowledge of the technical domain Starter-knowledge of Trados Studio 2014 Portsmouth Conference 2014

The debate that puts machine translation and professional translation on the opposite sides is misleading. The question isn’t about choosing ‘eitheror’ because those two solve totally different problems. Professional translation is the best choice when your goal is to get someone to understand your message. MT is a risky choice in those kinds of situations but is a more suitable tool when you want to understand someone else’s message. Multilizer, Translation Blog Portsmouth Conference 2014

Thank you! Iulianna. vander. Lek. Ciudin@kuleuven. be Frieda. Steurs@kuleuven. be

References • Thicke, Lori. The industrial process for quality machine translation. Retrieved from http: //www. jostrans. org/issue 19/art_thicke. php on 5 November 2014. • Guerberof Arenas, Ana (2009). Productivity and Quality in the Post–editing of Outputs from Translation Memories and Machine Translation. Localisation Focus 7(1): 1121 http: //isg. urv. es/library/papers 2009_Ana_Guerberof_Vol_7 -11. pdf • Guerberof Arenas, Ana (2013) What do professional translators think about postediting? . The Journal of Specialised Translation 19: 75 -95. http: //www. jostrans. org/issue 19/art_guerberof. pdf • O’Brien, Sharon (2002) Teaching Post-editing: A Proposal for Course Content. Proceedings of the 6 th EAMT Workshop on “Teaching Machine Translation”. EAMT/BCS, UMIST, Manchester, UK. 99 -106. http: //mt-archive. info/EAMT-2002 OBrien. pdf • Language Technology at Autodesk. Translation and Post-editing Productivity (2011). http: //langtech. autodesk. com/productivity. html • Doherty Stephen & Gaspari Federico , QT Launch. Pad Project (2014) Effective Post. Editing in Human and Machine Translation Workflows. http: //www. galaglobal. org/blog/2014/effective-post-editing-in-human-and-machine-translation-workflows/ Portsmouth Conference 2014

References • O'Brien, Sharon and Moorkens, Joss (2014). Towards intelligent post-editing interfaces. In: FIT XXth World Congress 2014, 4 -6 Aug 2014, Berlin, Germany. Retrieved from http: //doras. dcu. ie/20136/ on 5 November 2014. • • • Mate. Cat: http: //www. matecat. com • Patenttranslator blog: There Are Many Ways to Commit Suicide, But Post-Processing of Machine Translations Would Be a Horrible Way To Go (2014) : http: //patenttranslator. wordpress. com/2014/10/22/there-are-many-ways-to-commitsuicide-but-post-processing-of-machine-translations-would-be-a-horrible-way-to-go • Translationtribulations. com SDL: http: //www. sdl. com/products/sdl-trados-studio/ Kilgray Technologies: http: //kilgray. com Mem. Source: http: //www. memsource. com e. Mp. Ty Pages Blog: Improving the MT Technology through Translator Dialogue. http: //kv -emptypages. blogspot. nl/2014/05/improving-mt-technology-to-translator. html Portsmouth Conference 2014

References • Plitt. M, Masselot F. A productivity test of statistical machine translation post-editing in a typical localisation context. Retrieved on October 21 from http: //ufal. mff. cuni. cz/pbml/93/art-plitt-masselot. pdf • De Palma, Don. Post-editing in practice: http: //www. tcworld. info/e-magazine/translation-andlocalization/article/post-editing-in-practice/ • Asia Online, Fair-compensation of Post-editing MT: http: //www. asiaonline. net/EN/Resources/Articles/Fair. Compensation. For. MTPost. Editing. aspx. Retrieved on 27 October 2014 • Smets, Martine. Microsoft and the need for agile translations: http: //cdt. europa. eu/EN/news/Pages/default. aspx? CDTID=266 • Deliverables D 5. 1 Baseline Definition Results on KPI 1: Post-editing Effort: http: //www. matecat. com/deliverables/d 5 -1 -baseline-definition/results-on-kpi-1 -post-editing-effort/ • • • Apertium: http: //www. apertium. org/index. eng. html#translation ATA Chronicle (February 2013): http: //www. atanet. org/chronicle/feature_article_february 2013. php Example of a Translation Integrated Workflow: http: //www. asiaonline. net/EN/Machine. Translation/default. aspx? QID=21 Portsmouth Conference 2014