QUALETRA JUST2011JPENAG2975 With financial support from the Criminal

  • Slides: 38
Download presentation
QUALETRA JUST/2011/JPEN/AG/2975 With financial support from the Criminal Justice Programme of the European Commission

QUALETRA JUST/2011/JPEN/AG/2975 With financial support from the Criminal Justice Programme of the European Commission Directorate General Justice QUALETRA “QUALITY IN LEGAL TRANSLATION” WS 4 Testing, Evaluation & Assessment KU Leuven Antwerpen 16 -17 October 2014

Translation Evaluation Translation product (target text, the result of the translation process) Translation process

Translation Evaluation Translation product (target text, the result of the translation process) Translation process (think aloud protocol, eye tracking, key logging) Translation service (contact with the client, offering the quotation, invoicing, compliance agreements, complaints, etc. ) © Hendrik J. Kockaert & Winibert Segers

Translation Evaluation Translation product: methods Holistic method Analytical method CDI method (Calibration of Dichotomous

Translation Evaluation Translation product: methods Holistic method Analytical method CDI method (Calibration of Dichotomous Items) PIE method (Preselected Items Evaluation) © Hendrik J. Kockaert & Winibert Segers

Translation Evaluation Translation product: methods Research Reports: How do evaluators carry out the following

Translation Evaluation Translation product: methods Research Reports: How do evaluators carry out the following evaluation methods? (1) Holistic (2) Analytical (3) PIE

Research Reports Translation Evaluation Holistic & Analytical Methods Research Method A sample translation from

Research Reports Translation Evaluation Holistic & Analytical Methods Research Method A sample translation from English to Spanish of a UK robbery judgment was allocated to ten evaluators who received a translation brief (structured translation specifications) Shortly after submitting their assessment, the evaluators were asked to complete a questionnaire These case studies were carried out by UAH (Holistic method) and DCU (Analytical method)

Research Report Translation Evaluation PIE Method Preselected Items Evaluation Translation brief LT specific criteria

Research Report Translation Evaluation PIE Method Preselected Items Evaluation Translation brief LT specific criteria Essential documents EAW ECQA criteria © Hendrik J. Kockaert & Winibert Segers

Research Report Translation Evaluation PIE Method Case study KU Leuven carried out a case

Research Report Translation Evaluation PIE Method Case study KU Leuven carried out a case study where 46 level A professional translators from federal ministries in Belgium did a translation from French to Dutch The text contained 56 elements, including punctuation. Seven preselected items were used An intern corrected the translations using the analytical evaluation method, referring to the error categories of the ATA evaluation grid, without implementing the actual ATA format and criteria The standard error of difference between the scores obtained on the basis of the analytical method and the PIE method was 0. 177, which is not statistically significant

Sample Test JUGEMENT CORRECTIONNEL DEMANDE DE MISE EN LIBERTÉ DEBATS Avant l’audition de [PI

Sample Test JUGEMENT CORRECTIONNEL DEMANDE DE MISE EN LIBERTÉ DEBATS Avant l’audition de [PI 1] XXX, le président a constaté que celui-ci ne parlait pas suffisamment la langue française ; Il a désigné YYY, interprète inscrit sur la liste du tribunal ; l’interprète a ensuite prêté son ministère [PI 2] chaque fois qu’il a été utile. A l’appel de la cause [PI 3], le président a donné connaissance de [PI 4] l’acte qui a saisi le tribunal [PI 5] et constaté la présence et l’identité de XXX, dont [PI 6] il a reçu les déclarations [PI 7]. Maître ZZZ, conseil [PI 8] du prévenu, a été entendu [PI 9] en sa plaidoirie. Le ministère public a été entendu en ses [PI 10] réquisitions. Le prévenu a eu la parole en dernier. Le greffier a tenu note du déroulement des débats.

PIE: Case Study Statistics: Analytical Score vs. PIE Score 12. 0 10. 0 8.

PIE: Case Study Statistics: Analytical Score vs. PIE Score 12. 0 10. 0 8. 0 PIE Score (. . /10) 6. 0 Analytical Score 4. 0 2. 0 0. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 © Hendrik J. Kockaert & Winibert Segers

PIE: Case Study Statistics: T-Test P value and statistical significance: The two-tailed P value

PIE: Case Study Statistics: T-Test P value and statistical significance: The two-tailed P value equals 0. 1376 By conventional criteria, this difference is considered to be Not statistically significant Confidence interval: The mean of Group One minus Group Two equals 0. 268 95% confidence interval of this difference: From -0. 090 to 0. 626 Intermediate values used in calculations: t = 1. 5151 df = 40 standard error of difference = 0. 177 © Hendrik J. Kockaert & Winibert Segers

Research Proposals Translation Evaluation PIE Method Conclusion: No statistically significant difference between analytical scores

Research Proposals Translation Evaluation PIE Method Conclusion: No statistically significant difference between analytical scores and PIE scores PIE proves a justifiable evaluation method objectivity transparency equality © Hendrik J. Kockaert & Winibert Segers

Activities Translation Evaluation ECQA Test Design ECQA Test on three successive stages: (1) MC

Activities Translation Evaluation ECQA Test Design ECQA Test on three successive stages: (1) MC questions on legal knowledge (specifically EDs and EAWs) (2) MC and open questions on legal language (phraseologies, terminology, style, register), possibly including a recognition test (to recognise whether a proposed translation is correct or not); after passing each of the two previous stages (3) A translation test corrected with PIE

Deliverables Survey disseminated, analysed and survey report Legal Translation Product Quality Assurance (1) Three

Deliverables Survey disseminated, analysed and survey report Legal Translation Product Quality Assurance (1) Three research reports drafted (holistic, analytical and PIE method) (2) Strategy and methods have been decided for reporting on the evaluation methods (3) Submission of “Objective Translation Evaluation through PIE” (KU Leuven), 1 st International Young Researchers’ Conference on Translation and Interpreting (UAH, 7 -8/11/2013)

Deliverables No. 1 Deliverable name/type (a) Survey on the testing, assessment and evaluation on

Deliverables No. 1 Deliverable name/type (a) Survey on the testing, assessment and evaluation on the current legal translation practices in criminal proceedings in the EU (product, translator and service/process quality assurance) Format (b) Websites and manuals (updatable and online information on product, translator and service/process quality assurance) Target group (d) Translator trainers Legal practitioners Professional associations

When you evaluate legal translations, how important do you find the following? 100 90

When you evaluate legal translations, how important do you find the following? 100 90 80 70 60 Being able to differentiate between good and bad translations 50 An objective evaluation 40 The efficiency of the overall evaluation procedure 30 20 10 0 Very Important Neither Important nor Unimportant Not at all Important

Which evaluation method do you find most suitable for legal translations? 100% 90% 80%

Which evaluation method do you find most suitable for legal translations? 100% 90% 80% 70% 60% Evaluation by comparing the translation to the source text and highlighting any errors 50% Evaluation of the translation of preselected segments in the source text 40% Evaluation by means of a checklist or an evaluation grid 30% Evaluation based on the general impression of the quality of the translation 20% 10% 0% 1 (highest ranking) 2 3 4 5 6 (lowest ranking)

Re ab ilit ad rm Te xt te ol og y in rg et

Re ab ilit ad rm Te xt te ol og y in rg et ta e er s es gis t d re an yle f t h y o St ct n re co r fe r ne ss ns ra g t ni n ic at io m Id ea M ct xt te xt ce rre ic co st gu i Lin ur so e th o rg et d ta an xt es s t ln ith fu Fa te e rc so u n ee tw be e How important do you find the following? 120 100 80 60 40 Not at all Important 20 Unimportant 0 Neither Important nor Unimportant Important Very Important

Which are the most important issues in evaluating legal translations? 100 90 80 70

Which are the most important issues in evaluating legal translations? 100 90 80 70 Having a suitable source text for legal translation evaluation 60 Determining if the candidate translator passes or fails 50 Determining the severity of errors 40 Applying the same criteria to each translation 30 Avoiding subjective evaluation 20 10 0 1 (highest ranking) 2 3 4 5 6 7 (lowest ranking)

How can the evaluator justify the evaluation? 100 90 80 70 60 50 Agree

How can the evaluator justify the evaluation? 100 90 80 70 60 50 Agree 40 Neutral 30 Disagree 20 10 0 By showing that the By stating that the By making observations translation does not expertise of the on the nature and meet the requirements translation evaluator severity of any errors of a good translation guarantees an objective evaluation By completing an evaluation grid By evaluating the translation of preselected segments in the source text

Test Formats Three different testing formats: Translation test Revision test Recognition test

Test Formats Three different testing formats: Translation test Revision test Recognition test

Certification Procedure Prerequisite Only European Master’s in Translation Network (EMT) graduates may apply to

Certification Procedure Prerequisite Only European Master’s in Translation Network (EMT) graduates may apply to take the ECQA certification examination.

Examination The examination consists of 3 stages A candidate must pass each element of

Examination The examination consists of 3 stages A candidate must pass each element of each stage before passing to the next stage A candidate who fails one or more elements of a stage only has to repeat the failed element(s)

Examination The 3 stages of the examination are: Multiple choice questions: LTi. CP. U

Examination The 3 stages of the examination are: Multiple choice questions: LTi. CP. U 1 Legal Knowledge U 1. E 1 Criminal law and procedure U 1. E 2 Monolingual legal terminology (in both languages) Internet access is not permitted at this stage Open questions: LTi. CP. U 2 Professional aspects & LTi. CP. U 3 Instrumental competence U 2. E 1 Professional practice U 2. E 2 Professional conduct U 3. E 1 Information acquisition U 3. E 2 Legal terminology management Internet access is not permitted at this stage Translation: 2 x 250 -word texts, one for each element of LTi. CP. U 4 Translation U 4. E 1 Translation of essential documents (Directive 2010/64/EU) U 4. E 2 Translation of European Arrest Warrants Internet access is permitted at this stage

ECQA Skills Card Skill Card Type of Examination LTi. CP. U 1 Legal Knowledge

ECQA Skills Card Skill Card Type of Examination LTi. CP. U 1 Legal Knowledge U 1. E 1 Criminal law and procedure MCQ U 1. E 2 Monolingual legal terminology (in both languages) MCQ LTi. CP. U 2 Professional aspects U 2. E 1 Professional practice Open Questions U 2. E 2 Professional conduct Open Questions LTi. CP. U 3 Instrumental competence U 3. E 1 Information acquisition Open Questions U 3. E 2 Legal terminology management Open Questions LTi. CP. U 4 Translation U 4. E 1 Translation of essential documents (Directive 2010/64/EU) Translation U 4. E 2 Translation of European Arrest Warrants Translation

Translation Test maximum 2 x 250 -words PIE method

Translation Test maximum 2 x 250 -words PIE method

Test Format PIE Items Report a lost or stolen passport [1] You must report

Test Format PIE Items Report a lost or stolen passport [1] You must report the loss or theft of your passport as soon as possible, even if you don’t want to replace it immediately. You’ll need to complete a Lost or Stolen Notification form [2]. Find out how to get the form and what to do if your passport is lost abroad. If your passport is lost or stolen in the UK [3] You must report the loss or theft of your passport to the Identity and Passport Service [4] (IPS) [5]. This will reduce the risk of anyone else using your passport or your identity. To report your passport lost or stolen, fill in and sign a Lost or Stolen (LS 01) [6] Notification form [7] and return it to IPS [8]. The address is on the form. You should [9] report all passport thefts to the police. You’ll need the crime reference details for the LS 01 form [10].

Revision Test maximum 10 items maximum score = 10

Revision Test maximum 10 items maximum score = 10

Test Format Candidates have to revise the English translation; have to recommend corrective measures

Test Format Candidates have to revise the English translation; have to recommend corrective measures for elements in the English translation that are not correct. Vérifiez la date du document. Check the data of the document. A candidate can make three types of erroneous revision: 1) The candidate revises an element in the translation that was correct, and proposes, for instance, to replace ‘Check’ by ‘Control’ or by ‘Verify’; 2) The candidate does not revise an element in the translation that was not correct: in our example the word ‘data’, which has another meaning then the French word ‘date’ (‘date’); 3) The candidate revises an element in the translation that was not correct, but her/his revision is not correct, and proposes, for instance, to replace ‘data’ by ‘information’.

Recognition Test maximum 10 items maximum score = 10

Recognition Test maximum 10 items maximum score = 10

Example Candidates have to indicate if the proposed English translation is correct or incorrect.

Example Candidates have to indicate if the proposed English translation is correct or incorrect. Le témoignage de l’épouse de l’accusé n’est pas crédible. The testimony of the wife of the accused is not credible. correct incorrect

Test Format PIE Translation, Revision, Recognition After passing the three tests (translation, revision and

Test Format PIE Translation, Revision, Recognition After passing the three tests (translation, revision and recognition), the candidate will obtain a score with a maximum of thirty The proposed testing approach is a better guarantee for objectivity than the traditional translation test The score of the candidate does not depend on the personal appreciation and interpretation of the evaluator Another advantage of this testing approach is the time-saving aspect: The time you spend to develop the three tests is largely compensated by the short evaluation time

Validity of Tests Score 25 20 15 Recognition Revision Translation 10 5 0 Candidate

Validity of Tests Score 25 20 15 Recognition Revision Translation 10 5 0 Candidate 1 Candidate 2 Candidate 3 Candidate 4 Candidate 5 Candidate 6 Candidate 7 Candidate 8 Candidate 9

ECQA Overview Question Types Sample CLTi. CP. U 1. E 1 U 1. E

ECQA Overview Question Types Sample CLTi. CP. U 1. E 1 U 1. E 2 Learning Elements to be covered consistent with skill card Type of Examination Developer Professional competence Professional practice Open Questions Professional conduct Open Questions

ECQA Questions Samples LTICP. U 1. E 1. PC 2: The candidate has a

ECQA Questions Samples LTICP. U 1. E 1. PC 2: The candidate has a sound understanding of criminal procedure in the legal systems involved (e. g. levels of jurisdiction, legal structures, institutions, settings, parties). 1. A person who _________ a crime may be sentenced to imprisonment. Fill in the blank with one of the four choices: A B C D does is convicted renders performs

ECQA Questions Samples 1. 1. 1. LTICP. U 1: Legal knowledge LTICP. U 1.

ECQA Questions Samples 1. 1. 1. LTICP. U 1: Legal knowledge LTICP. U 1. E 1: Criminal law and procedure CLTICP. U 5. E 1. PC 5: The candidate masters the main domains and sub-domains of criminal law, especially the most frequent offences in essential documents and European Arrest Warrants, e. g. drugs, fraud and theft. 1. From the following list, choose one example of an offence against the person: A B C D theft smuggling assault possession

ECQA Questions Samples LTICP. U 1. E 1. PC 3 The candidate has a

ECQA Questions Samples LTICP. U 1. E 1. PC 3 The candidate has a general awareness of current legal issues and their development in the relevant countries. 1. From the following list, choose one example of a current legal issue of special concern in the EU cross-border criminal context: A Illegal sale and distribution of food products B Trafficking in human beings C Electronic surveillance by state security services D Non-payment of fines

ECQA Questions Samples LTICP. U 1. E 1. PC 4: The candidate is familiar

ECQA Questions Samples LTICP. U 1. E 1. PC 4: The candidate is familiar with the EU directives on legal translation. 1. A criminal suspect has the right to translation of (choose one of the following): A B C D Questions from the judge Prosecution statements Transcripts of evidence given in court by witnesses Other