The International Organization for Standardization (ISO), the body in charge of developing and publishing international standards, has published a new standard on evaluation of translation output.

ISO 5060 is the result of four years of work conducted by the ISO/TC 37/SC 5 technical committee for translation, interpreting and related technology, and the 18th document of this kind published by the unit (with a further seven being currently under development). Numerous experts on translation and quality evaluation formed part of the working group, with participants from more than 30 countries contributing to the creation of this new standard.

ISO certification brings an array of benefits for companies deciding to pursue it: increased sales and revenue, heightened efficiency, and improved quality of operations are usually expected as a result of certifying against a selected standard. Language service providers (LSPs) and others recognize compliance with ISO norms as a competitive advantage, allowing them to position themselves among top players in the industry.

The new ISO 5060 standard was published in February 2024. It “is applicable to translation service providers […], including individual translators, translation companies or in-house translation services, their clients and other interested parties in the translation sector, such as translator education and training institutions.” Aimed at providing guidance for human evaluation of the translation output, ISO 5060 can be applied to workflows involving both human and machine translation with or without subsequent post-editing.

The standard combines hands-on advice with the strategic considerations and general principles for translation quality evaluation. It is built on the analytic approach that involves a “bilingual examination of target language content against source language content while classifying any errors with respect to translation evaluation specifications and for the purpose of reaching a quality rating.” Focus on the translation requirements and treating any deviation from it as an error allows for a high degree of inter-rater reliability (IRR; guaranteeing that the results of evaluation don’t differ significantly depending on evaluator) and evaluation objectivity.

“The standard enables the user to produce reliable, trustworthy translation quality data and is therefore something completely different than MT predictive quality estimation.” — Christopher Kurz, member of the ISO/TC 37/SC 5 committee

ISO 5060 introduces a framework of concepts related to the evaluation of translation quality, and provides a set of requirements for all steps of the evaluation process, covering the pre-evaluation, evaluation, and post-evaluation phases. Error typology that has been harmonized with the first level Multidimensional Quality Metrics Framework (MQM), as well as guidelines on severity, weight, or the approach to repeated errors, form an important part of the standard.

According to the proposed typology, errors can fall into one of seven main categories: 

• Terminology;
• Accuracy;
• Linguistic conventions;
• Style;
• Locale conventions;
• Audience appropriateness; and
• Design and markup.

Each of the categories contains several sub-categories and every error can be classified as Critical, Major, or Minor, depending on its severity.

Additionally, the standard outlines the competencies of evaluators, enumerating aspects such as: understanding of the source language, fluency in the target language, cultural ability, knowledge of the domain, as well as formal qualifications, among others.

Annexes provide readers with a set of questions to guide them through the process of analyzing the evaluation needs and to help them develop a translation evaluation strategy and system. Practical recommendations on sampling, a detailed explanation of the concepts of cohesion and coherence, as well as ready-to-go scorecard templates are also included.

The next revision of the standard is planned for 2029.

Interested parties can obtain a copy of the ISO 5060 standard and the other standards related to language services from the ISO website.

Source: slator.com