29/04/2025

It’s Quality Time again. In a new episode of our series of interviews with experts, Eva-Maria Tillmann, Head of Quality Management at oneword, spoke to Dr Christopher Kurz, Head of Translation Management at ENERCON. In this interview, Dr Kurz talks about his remarkable journey to becoming the project manager of ISO 5060 for the evaluation of translations. An interesting discussion about standards in daily practice, and the concrete benefits for clients even as AI applications become more popular.

oneword and ENERCON have enjoyed a strong partnership since 2018. oneword is one of ENERCON’s translation service providers, and Eva-Maria Tillmann and Dr Christopher Kurz are both actively working on industry standards on the committee for translation services. Dr Christopher Kurz is a graduate translator (English and Italian with a focus on mechanical engineering and construction) with many years of experience not only in translation, but also in project and translation management at translation service providers and large corporations. He is now head of translation management at ENERCON, a global manufacturer of onshore wind turbines. He has been active at DIN and ISO since 2011 and is project manager for the ISO 5060 standard (Evaluation of Translation Output).

Eva-Maria Tillmann (EMT): Hello, Christopher! It’s great that we have the chance to discuss our ideas here today. We’ve known and respected each other highly for many years. We first met at the 2016 tcworld conference. In preparation for my presentation on translation quality and ISO 17100, I read your doctoral thesis entitled ‘Translational Quality Management’ and we got talking about it. I think that was when you first suggested that I could work at DIN.

Dr. Christopher Kurz (CHK): Exactly.

EMT: I only became involved with DIN in 2018, when you invited me as a guest to a meeting of the Translation Services Committee. Since then, I’ve been working there with great enthusiasm and, together with the other experts from the industry, we’ve worked on a number of standards projects that we can be really proud of. One of these was ISO 5060 for the evaluation of translations. You took on the project management for this standard.

Tell us, how did you get into standardisation work yourself? Was it because you were already working on the topic during your doctorate or had you actually already been active at DIN and ISO for a long time?

CHK: Yes, it’s a long story. Eva, first of all, I’m delighted that you’ve invited me here. I think this is a wonderful opportunity for us to look back together at standardisation and ISO 5060. To answer your question, I’ll go back to my personal background. During my studies in Leipzig, I got a pretty good all-round overview of the subject of translation, including defective source texts, terminology, tools, processes, quality management, etc.: in other words, everything that influences translation quality. While I was still a student I also worked as a freelancer for end customers from spring 2000. After completing my studies and working at a six-person agency in Leipzig, I joined SDL in Munich in 2006, where I looked after major clients and important customer projects. I was interested in doing a doctorate even back then. That was in 2007, and it quickly became clear that I found the topic of quality fascinating, as well as translation errors, the consequences, evaluation and quality assurance. As lead translators at SDL, now RWS, we were regularly evaluated by our colleagues and we also evaluated them. During this time, I learned a lot about what translation quality is and how it comes about.

EMT: I see.

CHK: I continued to be interested in the topic, but I was unable to pursue it with my employer at the time. In 2010, I moved to BMW to work on the manufacturer side. I had already started my doctorate on Translational Quality Management with Professor Klaus-Dieter Baumann at IALT in Leipzig in 2009 and as part of my dissertation I wanted to analyse DIN EN 15038, the predecessor standard to ISO 17100. At first, I was opposed to the standard because I thought: “What rubbish, you can’t stipulate all of these things and I’ve just experienced in practice that it works quite differently.” I thought maybe I could disprove the standard – that was actually my very first intention for my doctorate. I then started writing and, based on my practical experience, developed my own quality-orientated translation process and compared it with the standard. I also contacted DIN as part of my research and was told: “Yes, DIN EN 15038 is currently being revised, and we’re about to set up a whole new committee for it. Would you like to be involved?” And that, in November 2010, was ultimately how I first got involved in standardisation work. Then, in spring 2011, I attended the first meeting of the current standardisation committee for translation services and realised that standards have a lot of good things to offer, provide a lot of good guidance and that the work we’re doing is very useful. I stayed on and am currently one of the last two of almost ten committee members who attended the first meeting for interpreters and translators at DIN in 2011.

EMT: Then you really were there right from the start! I’ve been responsible for our certifications at oneword since 2012, so I’ve been intensively involved with the standards in our industry, but only as a user, not in standards development as such. That’s why I was so interested in helping to shape the standards myself when I was invited to do so. I’m sure it was similar for you. Neither of us would have come up with the idea of actively working on standardisation on our own, but we immediately found it exciting!

CHK: Yes.

EMT: I’d now like to know how you came to work on the evaluation of translations. You then actually became the project manager for ISO 5060, a standard that covers this subject, which means that you have a special connection to it.

CHK: This connection comes from both my doctorate and my professional experience as a translator. Over the years, I’ve come to realise what makes a ‘good’ translation: in essence, it’s about the quality concept from ISO 9001. There are a lot of ideas and theories about translation quality, but when you’re in the field, you actually realise that quality means a great deal – and very little of it is written in books. This includes compliance with terminology, using high-fuzzy matches without rewriting them, managing tool settings such as match value limits, options like auto-propagated fuzzies and the fuzzy match value in term recognition, and the TMs and source texts. At ENERCON, we – as the client – also look at who works for us and how much we outsource to which external partner – because we work according to metrics. And of course we also look at the overall process, including quality management with regular evaluation. Quality is an incredibly extensive conglomerate of so many components; it’s incredibly complex and is made up of so many factors that it truly interests me. If you manage everything and keep all the balls in the air, so to speak, it’s called translation management or quality management.

EMT: So the first time you encountered evaluation was when you were working for a translation service provider?

CHK: Exactly. I was assessed as a translator at SDL/RWS according to the LISA principle. I then introduced a similar quality assessment system for my project in the relevant department at BMW in 2011, and we then used it to assess commissioned translations and service providers. Because the bottom line is: as a manufacturer, you pay for something and want to get back a product that corresponds with what you ordered. Assessing this objectively is part of the resourcing process, because the translator or translation service provider can only make effective adjustments with the help of an evaluation.

EMT: I understand. I already know you also introduced this at ENERCON, because we regularly receive your evaluation results as feedback.

CHK: Yes, exactly! If you base your business decisions for the translation process on key figures, then these key figures should actually be valid, and that can only be achieved through objectivity. And that was the basic idea behind ISO 5060 right from the start. We wanted to make translation quality objective both at ENERCON and in the ISO working group and we were able to successfully implement the ISO 5060 principle in our day-to-day work at ENERCON.

My team has been working together on this translation quality management process and we’ve realised that the clearer we are in defining our requirements (this is also a key part of the new ISO 11669), the clearer we are about what we want and the tighter we make the process guidelines, the fewer errors there will be in a translation. And then this work becomes repeatable, it can be planned and it loses its lack of controllability. And that’s the beauty of ISO 5060: it requires professional distance, i.e. personal taste must be put to one side in the evaluation and a translation must be evaluated as objectively and neutrally as possible. If you create a tight framework, you can check everything properly. Then, everything that complies with the framework is compliant, and everything that doesn’t is non-compliant, meaning that it’s an error.

EMT: Did you introduce the evaluation system before ISO 5060 was in progress or did you introduce it at the same time as the standard was being developed?

CHK: A lot took place at the same time. The four years of work on the standard and the revision of our TQE matrix (note added by oneword: Translation Quality Evaluation) were also mutually beneficial and happened during the same period of time. A lot of theory has been used in practice and a lot of practice has flowed into theory – both from us and from the ISO 5060 working group, with participants from a wide range of sectors and from over 30 countries! The working group was great because real experts from client companies, service providers, freelancers, associations and universities played their part. These different stakeholders presented a very diverse picture and also provided a lot of different input. On the one hand, we had to reach some compromises, but, on the other hand, interestingly enough, it also became clear that we were going in the same general direction.

EMT: You said it: ISO standards are always the end result of many discussions and of international consensus. You had to make some compromises as project manager. Are you happy with the result? In your opinion, is the standard suitable in practice?

CHK: Definitely. I’m very satisfied with the result and I’m particularly pleased that we’ve kept all stakeholders on board. Even though we started from very different directions, no stakeholders left, and that was absolutely excellent.

EMT: Does your evaluation system at ENERCON actually correspond to the standard now or have you deviated from it?

CHK: We’ve adapted some minor details, such as the definition of severity levels, but our evaluation scheme basically corresponds to ISO 5060. Before working on the standard, we also met with the MQM group (note added by oneword: Multidimensional Quality Metrics) and had a very intensive, very constructive discussion, with the result that MQM and ISO 5060 don’t contradict each other. So for someone who already evaluates according to MQM, it’s straightforward to introduce ISO 5060.

EMT: By the way, I also think that the standard is very broadly applicable. Ultimately, we translation service providers are also clients and purchase services, which means that we can also use ISO 5060 to evaluate the individual process steps that we subcontract. And I also think the standard is a great tool for this. You’ve already touched on the benefits of evaluating translation services. Would you perhaps like to go into a little more detail about the extent to which this actually adds value to the translations you commission and what you get back from your service providers?

CHK: With pleasure. Each evaluation contains data points, for example which errors occur how often and with what degree of severity. These data points can be converted into graphics: the error patterns are visualised, so to speak. This allows you to see exactly which external partner has problems in which error area and how key figures and main error points are developing over the years. Subsequently, you can make selective and targeted adjustments to these main error points.

EMT: Can you give us an example?

CHK: For one external partner, it may be style in Spanish, in French it may be grammar and in English it may be mistranslations, i.e. content errors or omissions. For the next partner, mistranslations in English may be no problem at all, but terminology might be an issue instead.
This means that external partners can be precisely managed because the evaluation results can be used to tell them exactly where their weak points lie. Although this is downstream at the moment, the service provider can also initiate measures through a Kaizen process to reduce these errors or prevent them from occurring in the future. This allows you to work on the cause of the error. And then you truly have a quality management process and get ahead of the problem. You’re no longer running behind, but steering and directing – you’re managing the situation.

EMT: Does this mean that you were able to reduce risks by improving the quality in general?

CHK: Yes, it’s an effective means both of minimising risk and increasing customer satisfaction. And the great thing is, of course, that ISO 5060 can be used universally. ISO 5060 is open to any setting: you can evaluate human translation, with or without a TM, and you can also evaluate MT output, whether unedited or post-edited, because comparing a translation with the requirements is at the core of ISO 5060. This also clearly reflects the ISO 9001 concept. And this is an excellent way to ensure that you get exactly what you ordered.

EMT: It’s also always important for us as service providers to improve processes and align them to meet all customer requirements. If clients have worked out the specifications as precisely and concretely as you have, then that’s a real benefit for us too.

That type of discussion is ideal, but by no means does it happen with all clients. Of course, it’s not mandatory, but as you say, you want to minimise the risk for your own company and satisfy the end customer, and if something isn’t right in the process, then it needs to be addressed.

CHK: Yes, exactly. We also developed our requirements for the translation process using ISO 17100. Although we’re not certified to ISO 17100, we work almost in compliance with this standard, and it has also helped us enormously in developing the service description from the framework agreement.

As Translation Management, we were able to learn a lot from the processes of other departments and copy sub-processes, enabling us to truly industrialise the translation process: away from the purely operative language department or the ‘language uncle’ to a fully integrated and formative sub-process in the value chain. Evaluation and the key translation figures derived from it also played a major role in this.

EMT: Let’s move on to artificial intelligence: do you think AI can check and correct error categories from ISO 5060?

CHK: Florian Faes from Slator said so well that ISO 5060 is a toolbox. We provide users with a toolbox, and they can use all the tools in it or just some of them. Certain error categories can certainly be checked by AI, for example style guidelines, terminology, grammar and spelling. The current version doesn’t cover AI or AI evaluation, but this is because the standard was developed before ChatGPT came out and the whole hype around AI started. The Final Draft International Standard was put to the vote in autumn 2023.

AI can certainly check certain requirements. That’s a fact – no matter how you look at it. What it can’t handle – and humans are simply unbeatable here – is semantics, i.e. meaning, mistranslation, omissions, additions. And to be able to judge whether something is factually incorrect, you have to know the product. We manufacture wind turbines and the top requirement that we place on our translators, both internally and externally, is: be familiar with our products in detail, then you’ll understand what it says and what it means and then no mistakes will be made. Product knowledge is the absolute key. I really place a lot of value on this: when it comes to my in-house colleagues, our external partners and also myself personally. I want everyone who works for ENERCON in my area of responsibility to be genuinely interested in our turbines and our technology. It’s a problem if they’re not.

The issue of mistranslations is also given far too little attention in the industry because it can’t be accurately mapped using AI and sold as a tool. The scene at the moment is so incredibly AI-centred and completely blind to mistranslations, I think. I remain firmly convinced that semantics and meaning cannot be reliably represented in a vector space. Only a human can really ‘understand’ something. The machine or AI only calculates probabilities. You could also say that AI rolls the dice quite reliably. With good results. But it’s rolling a dice: it doesn’t understand. Don’t be fooled by the whole discussion around AI tools and AI processes. The business interests of tool and process providers are clearly behind it. But that would be a whole different conversation.

EMT: That’s probably true; the issue of evaluation is also affected by this hype and there are many new products and tools for this.

CHK: I’m glad you’ve brought that up: it gives me the opportunity to address the issue of quality estimation. Quality estimation isn’t an objective assessment of translation quality. A quality estimation only indicates, to a certain degree of probability, which MT matches will require post-editing and which will not. We’ve received initial feedback that automated quality estimation captures the extremes well, i.e. matches that are completely wrong or completely correct, but that the entire ambiguous grey area is not reliably assessed.

EMT: You’re right, this is one of those new topics that you have to watch. ISO 5060 is still relatively new. Where do you think the path will lead, regardless of the AI issue?

CHK: Standards are reviewed and revised every five years if the international community sees a need. The next systematic review will take place in 2029. The topic of AI, for example, could then become part of the standard.

In any case, I can see this standard having a big reason to exist for a long time. After all, unfortunately, most clients don’t bother to check the content of their translations. Terminology is straightened out and grammar and spelling mistakes are corrected, but there’s unfortunately no eye for detail and content. And that’s precisely what AI tools simply can’t check. I’d like to see ISO 5060 established in more companies, but also in universities. Students and lecturers alike must learn that translation evaluation must not be arbitrary but strictly orientated towards the requirements of the translation.

The standard really does offer a lot of guidance – for instance, it provides questions on implementation and three scorecards that can be used immediately. Incidentally, the first CAT tool companies are already integrating the scorecards from ISO 5060 into their evaluation features, which of course makes me happy. This will make the standard even better known.

EMT: This is a good example of how standards have a real practical relevance.

CHK: I’m also fully behind it and see the benefits in my day-to-day work at ENERCON. Translation quality brings peace of mind! We also receive a lot of positive feedback from various stakeholders, for example universities, the EU and various associations.

EMT: I’m not surprised; the standard has turned out really well! Christopher, it’s great that you were able to join us for Quality Time. Thank you very much for such a fascinating discussion!