The latest ToolKit newsletter by Jost Zetzsche contains a particularly interesting selection of free and premium content, the latter including what is probably the best overview I've seen yet of the innovations in Kilgray's memoQ 5. However, one of the most intriguing sections was the review of MemSource, in which a model for evaluating and compensating translation content produced with MT assistance was mentioned:
" ... what's the new paradigm that is being proposed (and used) by this tool? It's how machine-translated matches are optionally analyzed and potentially charged. If, they say, we can evaluate TM matches by the source (a perfect match is an identical source and a fuzzy match a similar source), we should look at MT matches by the target.What?Well, if there has been no change between a machine-translated target segment and a final target segment, it should be viewed as a perfect MT match and charged accordingly. If the changes are minor, it should be viewed as a fuzzy match, etc."
Up to now, the only really plausible proposals I've seen for compensating MT post-editing or MT-assisted translation have involved hourly work; those I've seen for piecework (word, page or line rates) have been exploitative at best. However, should one engage in the dubious practice of applying machine translation for language services, this paradigm is the most reasonable I have seen yet for assessing the "quality" of the output as seen by the degree of modification required. If for a particular purpose the MT output requires nearly a complete rewrite, it would essentially be paid as a completely new translation from scratch. If the MT output is in fact of somewhat useful quality, this provides a good quantitative means of assessing that and figuring the charges.
I will not say that I support drinking from the poisoned well of machine translation output and affecting the quality of one's other work in the same way that editing monkey work and reading trashy tests might do, but if one were to engage in such a foolish endeavor and join the lemmings surging toward the special interests' "future", this is at least an economic model worth discussing.
Of course, those who favor fixed rates for simple-minded budget planning will still want their straight word rates based on possibly untenable assumptions of quality. But the MemSource paradigm, which could be adopted easily by others or implemented with simple quantitative comparison tools (possibly even looking at an overall percentage change in structure, which might include rearrangements of blocks of text), is, as I see it, the first reasonable, practical suggestion for skipping over the nonsense of up-front "quality metrics", getting to work and letting the chips fall where they probably should.
Hi Kevin,
ReplyDeleteEven if post-editors were paid better rates (even if they were paid rates ten times their normal translation rates), there is a weak leg [*] of post-editing which consenting editors would be helping to strenghten: the greedy machine needs slave post-editors to continue increasing the corpus. (We all know about the lethal joint venture: MT + TM). So at this point it does not matter if you are offered huge rates, as the mere concept of post-editing is negative in essence, because the more the machine is fed, the less work they/we will have. 500,000 slave post-editors needed in 2011. How many in 2012, and in 2013?
Have a great week!
Au
ps: I am not questioning PE software which WE (translators) might use to speed up/improve OUR work, but PE jobs assigned by agencies. Yep, I'm aware of current limitations, but this should be priority issue for translators worldwide and the associations representing us/defending us.
Ingenious, Kevin, certainly. I subscribe to the Toolbox but skipped over this item. One problem that is apparently common to all payment schemes for this technology: it sets up skewed incentives. The translator will have the incentive to change as many segments as possible to raise his average per word rate. Whatever remuneration scheme is chosen, ultimately the agency owners will have to trust their freelancers, or, alternatively, very large PE shops will just have to ram through their remuneration scheme in a "take it or leave it" fashion. Still, it is an interesting (and thorny) problem, isn't it?
ReplyDelete@Miguel: I don't see the target text approach here as being any more problematic than other approaches to target text compensation. It's a simple enough matter to review the type and extent of changes to see if they are reasonable given the instructions of an assignment. Really, games can be played on both sides with any mode of compensation if parties lack good faith. The real solution here is to work only with people you can trust.
ReplyDelete@Au: Haven't you heard the plan? Redundant PEs will be post-processed as soylent green as part of the sustainable future economy.
It may surprise many people, including translators, but the number of changes made during post-editing is not proportional to the time spent, at least based on the results of our extensive translation productivity tests. There are people who are much faster at PE than translation from scratch although they make many changes, as there are people who make only few changes but still gain little time. So if you expect your payment to ultimately reflect the time you spend on a job, edit distance is probably not the way to go. Also, I really dislike the idea that as customers we should mess around with how translators do their work, I mean like instructing them to only make the fewest changes possible.
ReplyDeletePersonally, I believe that time- or job-based payment is the way to go in our industry, not only for/because of PE and the trust issue Miguel points out, but because paying per unit is just not appropriate for intellectual work performed by experts. This won't necessarily mean that translators will be paid more, but I would expect that going away from wordcounts and other mechanical translation concepts will help re-balance some of the oddities of our industry.
Although here at Autodesk we own our own MT solution (for some reason, in the translation industry innovation generally originates on the customer side, which is yet another oddity), I believe that in the longer term, it should be the translators who own the choice of the appropriate toolset for a given job, where that choice should of course include post-editing of MT, just like it may include dictation software and other technologies yet to emerge. At least for larger jobs, translator can then quote based on their recommended setup.
Good insights, Mirko. I would have thought of edit distance to be more representative of effort spent. When I was a PjM, we used to pay review projects hourly, the metric being 1500 words reviewed by hour (of course, quote could be revisited if quality was too bad). Maybe it'd be useful to agree on a similar standard for PE, like 750 post edited words/hour.
ReplyDeleteOn the giving tools back to translators part, I try to make an effort to let translators know what systems are available to them and how they can take advantage of them. So far, it seems translators think only of MT in terms of Google Translate and Bing, which have issues in terms of confidentiality and customizability. For MT to be of any use to translators, systems need to be customized. There are several open-source tools that can be tried before going for commercial software. I personally recommend Moses and Apertium. One is rule-based and the other is statistical (can be trained using TMs as corpora). Depending on each translator's work (kind of texts, volumes), it might make sense to invest time in getting acquainted with these systems and use them to get draft translations. They can also be integrated with CAT tools. Can provide more info to anyone who's interested.
Mirko said: "I believe that in the longer term, it should be the translators who own the choice of the appropriate toolset for a given job, where that choice should of course include post-editing of MT, just like it may include dictation software and other technologies yet to emerge."
ReplyDelete100% agree!
I was an in-house translator and post-editor for a large translation company that was one of the first ones to work with machine translation many years ago.
ReplyDeleteI did post-editing and also fed the machine to improve the output as much as possible.
We were paid the same hourly rate for translation or post-editing. There was no need to establish a scale since it took time to correct sentences, even the ones with very little change needed as the words had to be moved around.
The company saw the advantage in the general speed for large projects. We, the translators, had to struggle a lot with machine translation issues to produce a decent text fast.
It is true that you can get good outputs, but that means lots of time needed to prepare the machine and feed it so it will give you sometimes a correct translation or one that needs retouching only. But you can also get many absurd sentences and even some that give you just the opposite meaning. (Yes, machine translation can give you the opposite meaning in the target language when it cannot handle certain structures or when it has not been trained to do so.)
Very often it is faster to retype the sentence or the paragraph rather than jumping every couple of words to fix or delete MT mistakes.
Agencies usually do not want to spend time feeding the machine as they see it as waste. That increases the work of a post-editor, of course.
To make things worse, remember that many executives in some translation agencies have no idea about all you need to know and do in this profession. They usually do not even speak a second language, so they have no clue and anything written seems OK to them.
CAT tools and MT should be used without cutting down any rates since they help on one hand but present many issues that delay the process or require extra work on the part of the translator, post-editor and even agencies. Memories need to be updated and improved, and the same is true for the language repository in a MT project. Translators should not be “punished” for working with sophisticated tools. These tools require more expertise than a simple text you translate in Word.
Hi Kevin,
ReplyDeleteI am here from Jost Zetsche's blog. Your's are very interesting economist's considerations: how much and in which manner one should pay to rely upon a translation (a translator). Following this logic let's pay less to those miners who extract worse quality oil (it's in the Arctic).
I am a translator and live off, for three decades, on these earnings only and exclusively, no rent, no other income. And I may say that my time is my time, whether I translate raw text or review pre-translated one. By which method (human or machine) it was pretranslated, that doesn't matter. Sometimes the machine translates much better (e.g., the lists, etc.), than native and graduated human being. However, a translator only may determine this for any specific job, whether the pretranslation has been done to his satisfaction (taking into account personal translator's factors as well (tired, fresh, etc.)) or not. My point is that the proven translator should exist as such, and once relieved from the pressure of various and variable factors, he/she will produce much more and better than based on calculations on a specific job. You will have only to check regularly his/her translator's ability, that's all. And continuously and steadfastly improve the performance of MT-tools, without distracting your mind by the payment issues.
Wish you all the best,
Gennady Lapardin