Basic Rules of Engagement Desired output of workshop

Basic Rules of Engagement Desired output of workshop teams Scores Justifications Computational procedures Pitfalls

Desired Output of Teams On last two days n You will be presenting your evaluations w Methodology w Basis for methodology w Results w Analysis of results n Doing a short write-up of results MT-Summit Workshop Strong-Arming

Winning Scores Really useful evaluations n n Are objective Are replicable w Not resource intensive w Scalability is keen n Are meaningful w To MT providers / developers w To MT users w To linguists / NLP researchers n n Fit into the framework or inform changes Can be applied across language pairs / families

Justifications Relationship to ISLE framework n Or reasonable argument for changing it Relationship to linguistic / NLP research Relationship to “real world” n Task-based evaluation

Computational Procedures Should speak to automating (as much as possible) evaluation process Should be mathematically rigorous Must reflect available data tools where possible n n Caveats – bringing your own data Request for sensitivity of current data sets

Pitfalls We only have a week This is a workshop not a holy war MT will not take the jobs of translators n n Evaluation with respect to goal of output use, not the holy grail Evaluation with respect to continuum of use (which pieces are best machine / human)