Predicting What MT is Good for: User Judgments and Task Performance
Kathryn Taylor and John White
Litton PRC, 1500 PRC Drive, McLean, VA 22102, USA
{taylor_kathi, white_john}@prc.com
Abstract.  As part of the Machine Translation (MT) Proficiency Scale project at the US Federal Intelligent Document Understanding Laboratory (FIDUL), Litton PRC is developing a method to measure MT systems in terms of the tasks for which their output may be successfully used.  This paper describes the development of a task inventory, i.e., a comprehensive list of the tasks analysts perform with translated material and details the capture of subjective user judgments and insights about MT samples.  Also described are the user exercises conducted using machine and human translation samples and the assessment of task performance.  By analyzing translation errors, user judgments about errors that interfere with task performance, and user task performance results, we isolate source language patterns which produce output problems.  These patterns can then be captured in a single diagnostic test set, to be easily applied to any new Japanese-English system to predict the utility of its output.
The Machine Translation (MT) Functional Proficiency Scale Project
Machine translation (MT) no longer stands alone.  The contemporary context for MT is as an integral part of an end-to-end text-handling process, for which the trend is toward less human intervention between the process stages.  Documents in any form, including hardcopy, are funneled automatically into a corpus of on-line information made available to monolingual users to work with in their own language.  So MT has evolved under the presumption of less human intervention: what is needed now is a more precise judgment of the capabilities of an MT system to produce output suitable for whatever the next step is in a given user’s text-handling process.
Monolingual analysts or other information consumers, our target user pool for development of a metric predictive of MT capability, typically perform one or more text-handling tasks using translated material.  In this paper we refer to them as users or analysts.  Each type of text-handling task (e.g., filtering, detection, extraction, summarization, publication) requires translated text input of a certain quality.  For certain of these tasks there is a wide range of acceptable quality, as judged by the accuracy and completeness of MT output, while others require near-human accuracy and fluency.
What is needed is an ordering of the inventory of text-handling tasks, ranging from those with substantial tolerance to poor quality output to those that can only be performed with high-quality output.  Also needed is a measure of an MT system’s ability to produce suitable output for each of the “downstream” text handling tasks in the inventory, regardless of whether it is a human or automated process.  The FIDUL MT Functional Proficiency Scale project (MT Scale, for short), now in the data collection phase, is working toward producing a reusable, efficient and meaningful predictor of the text-handling tasks that an MT system’s output will support (White and Taylor, 1998).
Evaluation of MT remains a fundamental issue in the field (Church and Hovy, 1991; Dostert, 1973; Pierce, 1966; Van Slype, 1979).  MT evaluation has proven costly and subjective: because there is no one “right” translation, there is great difficulty in creating a reusable, extensible ground truth for evaluation from which a judgment of MT quality can be made.  Also, each stakeholder in machine translation evaluation (translators, information consumers, managers, researchers, etc.) needs a different view into MT.  Compounding the issues are the radically different theoretical approaches and language pairs, making a single-step, multi-purpose MT evaluation process difficult to envision.
The series of black-box evaluations of the Defense Advanced Research Projects Agency (DARPA) MT initiative within the Human Language Technology (HLT) Program in the mid-1990’s (White and O’Connell, 1994; 1996) had as a goal the evaluation of the progress of the core algorithms of the sponsored systems.  The diversity of theoretical approaches, language pairs and end-use presumptions were factored out by using a large sample of monolingual end users as raters.  The cost was high: in order to counter the inherent subjectivity of judgments about translation, a large number of translations, controls, and decision points had to be maintained.  However, the results of this large-scale effort (in terms of the fluency, adequacy and informativeness measures developed) can be reused in this study.  For MT Scale, the evaluation question is framed in terms of the user task performance that a particular level of MT output quality allows.
Ideally, an MT evaluation method should be readily reusable, with a minimum of preparation and participation of raters or subjects.  To accomplish this, a corpus for which judgments of MT output quality had already been made was reused, namely, the translations of the DARPA MT evaluation series.  This corpus was already rated for adequacy, fluency and informativeness.  The complete corpus consists of several translations each of approximately 400 newspaper articles, originally in French, Spanish, or Japanese.  A great advantage to this corpus is that, for most of these articles, two professionally translated versions were prepared.  Two to five machine-translated versions of each article exist, generated either by sponsored research systems or by commercial MT systems.
We will make use of existing resources, such as the DARPA corpus, as accomplish the objectives of the MT Scale project, which are to:
•	identify the text-handling tasks that users perform with translated material as input 
•	discover the order of text-handling task tolerance, i.e.,  how good a translation must be in order for it to be useful for a particular task;
•	analyze the translation problems (both linguistic and non-linguistic) in the corpus used in determining task tolerance; and
•	develop a set of source language patterns which correspond to diagnostic target phenomena.
Developing the Task Inventory
Three sources of information for formulating the inventory of tasks that analysts perform using translated material were used: (a) interviews with former analysts (b) unclassified panel discussions during which officials of the intelligence community stated analysis requirements and working analysts described their tasks, and (c) brief interviews with individual analysts, with the cooperation of their senior management.
Interviews with former analysts.  Before approaching working analysts about the type of tasks they routinely perform with translated input, project staff interviewed former analysts about their uses of translated material.  The interviews with former analysts provided a very preliminary concept of the task inventory and its ordering.
Unclassified panel discussions with officials and working analysts.  At government-sponsored program reviews and conferences, analysts often express their “wish lists” for tools and working aids. In defining their needs, they often describe the tasks that they do, and the volume and type of information that cross their desks each day.  These discussions provided a general sense of the volume of translated information presented to users and what they needed to do with it.
Brief interviews with individual analysts.  A telephone pre-interview survey was formulated to identify analysts by agency and the tasks each performed using translated material.  The interview is designed to capture user observations about the role of translation-supported tasks in their day-to-day work, and the quality of translation they perceive as necessary to accomplish those tasks.  In the telephone interview, users are asked which text-handling tasks (manual or automated) they typically perform.
The result is a comprehensive list of tasks which are performed using translated material.  The tasks identified by individual analysts determine which of the task-based exercises those analysts will perform at the time of the face-to-face interview.  At any time during the information-gathering process new tasks may be included in the inventory, and variations of a task category may be ordered within that task category.
Ordering the Task Inventory
To use the task inventory effectively, it is necessary to identify which text handling tasks can be done with relatively poor quality text, and which require high quality text.  Certain analytical text-handling tasks require more accurate and fluent material as input than others.  It should be possible to rank these tasks on a scale from least tolerant (high end of the scale) to most-tolerant (low end of the scale) of translation errors and omissions.  Table 1 shows a hypothetical ranking of tasks from least tolerant (Publishing) to most tolerant (Filtering).
A ranked order of text-handling tasks such as Table 1 implies that an MT system whose output can facilitate tasks on a particular point on the scale should also facilitate tasks lower on the scale, and is unlikely to facilitate tasks higher on the scale.  According to Table 1, an MT system that produces a fluent translation (publication quality output) will also support the capture of specified key information from the same translation (extraction).  An MT system whose output allows users to recognize that a document is of interest (detection) may not be suitable for capturing all specified key information (extraction), but will perform acceptably for filtering (rapid disposal of irrelevant documents).
Table 1. Preliminary Ranking of Text-Handling Tasks
Task	Description	
Publishing	Produce a technically correct document in fluent English	
Gisting	Produce a summary of the document	
Extraction	For documents of interest, capture specified key information	
Triage	For documents determined to be of interest, rank by importance	
Detection	Find documents of interest	
Filtering	Discard irrelevant documents	

As more interview data is gathered and further research performed, there is the possibility that certain tasks may be subdivided.  Based on a task description of variations of the extraction task provided at the TIPSTER 18-month meeting, shown in Table 2, Intra-task Ordering - Multiple Levels of Extraction, we became aware that users might draw distinctions between superficial name-spotting tasks and extraction requirements that require a more comprehensive understanding of a text.  This raised the possibility that a familiar class of task, upon closer examination, had separate incarnations that could be characterized as less tolerant (deep extraction) and more tolerant (shallow extraction) of MT output errors.
Table 2. Intra-task Ordering - Multiple Levels of Extraction
Type of Extraction	Description	
Deep	Event identification (scenarios):  the ability to identify an incident type and report all pertinent information	
Intermediate	Relationship identification:  (member-of, associate-of, phone-number-for)	
Shallow	Named entity recognition:  (isolation of names of people, places, organizations, dates, locations)	
The Face-to-face Interview
At the time of the face-to-face interview, users are asked to perform a variety of activities associated with their specific text-handling tasks, using translations from the DARPA corpus to determine, within the context of their individual ordinary activities, whether they could use a particular translation to do their jobs.  There are many issues (e.g., domain knowledge and experience) associated with eliciting this judgment.  Many such users have a functional mission (e.g., find information about grain production in Europe) rather than the performance of a text handling task per se.  The samples presented are general news articles, and are not directly relevant to the domain of their functional task.  Human factors effects can also bias the judgment.  For example, the user may feel obligated to respond positively when asked “can you do your work with this document” even when he/she actually cannot (M. Vanni, personal communication).
To address these issues, the interview includes three distinct exercise types:
•	The Snap Judgment Exercise -  indicating what texts are of sufficient quality that an analyst would be willing to process them further.  Users are asked to make snap judgments about a set of documents, separating out those which might be of good enough quality to enable them to do their jobs.
•	The Task-Specific Exercises -  performing a directed task on a set of documents.  Users are given a task that is similar to their ordinary text-handling task, and asked to perform this modified task on a set of translation documents.  For example, a user who performs filtering is told to set aside the documents that are definitely not relevant to a given topic.  The directed task validates the results of the snap judgment exercise, abstracting away from the domain issue (because the specific task is rather different from their actual tasks) and from several human factors issues.
•	The Rating Reasons Exercise -  helping to identify the translation problems which cause a document to be less useful than it otherwise might be for specific tasks.  This exercise is aimed at identifying translation phenomena that should be included in the MT Scale diagnostic test set by capturing the phenomena that disturb users most about the texts they have just tried to make use of in a task-oriented exercise.  For example, a user may cite lack of capitalization, transliteration and numerical errors when trying to use a text to perform an extraction task.  The significance of these phenomena would be noted for extraction, and source language patterns that produced the errors would become part of the diagnostic test for extraction capability.
Of course, no one group of users will generally require or have expertise in every one of the text-handling tasks that uses translated material as input.  Thus the ranked list will be a merged set over a variety of user groups.
The result of these exercises will be a characterization of the relevant text-handling tasks for the subject group of information analysts, ordered by their tolerance to the quality of MT output.  It remains to be established whether to expect a single ordering of the text-handling tasks (e.g., that document detection is always more tolerant than extraction), or a non-deterministic order (detection is sometimes less tolerant than extraction in different subject domains, or extraction requirements vary, etc.).  It appears, however that even a multi-path ordering, once described, will allow formulation of a scale, as long as there is convergence at either end (a reasonable assumption – topic filtering must always be more tolerant than technical editing).
The next paragraphs describe in depth the exercises that take place at the face-to-face analyst interviews: the snap judgment exercise, the specific task-oriented exercises, and the rating reasons exercise.
The Snap Judgment Exercise
The complaint is often made that MT output little resembles the target language.  This lack of resemblance frustrates a native speaker of the language to the point of not being willing to make the effort to try to understand it. The threshold of erroneous output, in the context of a particular downstream task, should indicate the tolerance of that task.  In the snap judgment exercise, analysts are asked to separate a set of 15 documents (selected from the DARPA Japanese-English corpus and pre-analyzed for errors) based on a superficial examination and their snap judgment of the overall quality of each document in the set.  As a guideline, they are told to spend as much time examining each text as they would skimming an article in a newspaper and deciding whether to read it in depth.  They are to divide the texts into three piles: 1) texts that they definitely thought they could process in connection with the task they do, (2) texts they might try to process (3) and texts they would not be willing to process further.  The analysts are not pressed for specific reasons for relegating a particular text to a particular pile, but any comments they volunteer are captured.
The main purpose of this exercise is to determine the tolerance level for machine translation output for individual text-handling tasks.  Users who perform the same tasks will be expected to have similar sortings of the texts into pile (1), (2) and (3).  It is expected that the difference in distribution of piles (1), (2) and (3) among the different tasks will help determine the tolerance of a particular task for a particular quality level of MT output and provide clues to the kinds of MT output errors that affect the use to which an output sample may be put.
The Task-Specific Exercises
In this series of exercises, users perform a task that is similar to one they do daily, using a set of translated documents.  The following are examples of some of the task-specific exercises.
The filtering task.  For the filtering task, each user is given a set of fifteen texts and given a very broad information category, such as crime, economics, transportation, etc.  One by one, the user is asked to categorize each text as being related to the information category, not related to the information category, or say that he/she is unable to tell (cannot be determined).  The results of this task will be reported using the measures of recall and precision, i.e., how many of the possible texts related to the information category did the user identify (recall), and how many of the texts the user identified as being related to the information category were actually related (precision).  The ground truth for the filtering exercise was determined by a three-member panel who looked at an expertly translated version of the text to determine if the text was related to a broad information category.
The triage task.  For the triage task, each user is given an ordered list of 2 to 4 narrowly described categories such as poisonings, foreclosures, airline industry deregulation, etc.  The user is asked to organize a group of fifteen texts in the same order as the list, and for texts of the same category, to order the texts by putting the most specific and information-rich texts first.  The results of this task will be reported using the measures of recall and precision, i.e., how many of the possible texts related to the more narrowly defined information category did the user identify (recall), and how many of the texts the user identified as being related to the more narrowly defined information category were actually related (precision).  The exercise includes elements of the filtering task exercise, but the triage task calls for more in-depth judgments of text content (amount and relevance of usable material) than the filtering exercise.
The extraction tasks.  Three levels of exercises are currently envisioned for extraction task, (1) one level will test the reporting of individual entities (names, places, organizations) found in a text, (2) a second level will test the reporting of relationships (person-to-place, person-to-organization. etc.) in a text and (3) the final level will test the recognition of a particular event type and the reporting of standard, pertinent information related to the event (e.g., for a bombing event, the location, date, time, type of device, persons injured or killed, buildings damaged or destroyed, group claiming responsibility).  Users will be given guidelines based on the Message Understanding Conference (MUC) Named Entity and Scenario template tasks.  The tasks will be scored according to the number of correct fills made from consulting a machine-translated version of a text as compared to the number of fills made by users working from the expert human translations of the same text.
The gisting task.  The bracketed version of the DARPA expert translation used to evaluate the adequacy of a translation will be excerpted so that only those brackets that contain information relevant to a summary of the document remain (a three-member panel reached a consensus of which brackets contain information relevant to a summary).  The user will be asked to apply the DARPA 1-to-5 adequacy scale ratings to indicate to what extent the information in the selected brackets is present in the aligned paragraphs of a translated version of the document. Low ratings will indicate missing/garbled information, and point to the source language pattern which produced the deficient portions of the translated document.
The publication task.  For the publication task, a user will be given excerpts of texts, one at a time, and asked to make any corrections to them that would make them, in the judgment of the user, publication-ready.  The user will also be directed to mark excerpts from a text that are publication-ready, or that need only minor corrections to be considered publication-ready in the user’s opinion.  The user is permitted to “give up on” a text at any point, and move on to another text. This would be a timed exercise, lasting 30 minutes.  The number of texts “given up on”, would be recorded, as well as the number and type of corrections made by the users to texts.
The Rating Reasons Exercise
This exercise is aimed at identifying the translation phenomena that should be included in the MT Functional Proficiency Scale diagnostic test set.
The rating reasons exercise is a variation of a study conducted to compare the criteria used by American and Japanese English-as-a-Second-Language (ESL) instructors to rate student compositions (Connor-Linton, 1995) The instructors rated compositions and then gave reasons for their ratings. The study demonstrated a correlation between  translation concerns and training, related in turn to differences between the Japanese and American instructors.
For the rating reasons exercise, a user is asked to assign overall ratings to texts from the DARPA corpus the analyst has just seen as part of a task-oriented exercise. Then the user is asked to list the three main reasons why the particular rating was assigned. As suggested by the Connor-Linton study, users should react to the material according to their training in the particular text handling task they perform.
As in the Connor-Linton study, reasons are not suggested to the participants, and the reasons may be positively or negatively expressed. All of the ratings reasons are collected and organized into categories by a linguist, and coded with the analyst’s task experience and the task exercise for which he/she was a participant. This allows the rating and reason data to be considered on a task-by-task-basis and on an reason-by-reason basis, as a single reason may have been cited by analysts in connection with more than one task.
The purpose of this exercise is to allow the alignment of particular output errors with the success or failure of specific text-handling tasks.
Tasks and MT Output Properties
If it is possible to predict the least tolerant text-handling tasks that a system’s output can facilitate, then we will also know that the output is sufficient for all of the more tolerant tasks. The same texts from which the ordering can be inferred also provide evidence of translation problems which indicate the boundary between acceptability for one text-handling task and another.
Developing the diagnostic test that will make that prediction involves identifying the correlation of corpus texts to the task hierarchy, distilling translation problems that appear to be “diagnostic” (i.e., appear to mark the difference between a text being at one level rather than a higher one), and then characterizing those translation phenomena in a compact pattern for the ultimate diagnostic test.
The phenomena encountered are categorized in accordance with established contrastive principles of Japanese and English. Use of the pedagogical models as applied to authentic text has the advantages of exhaustiveness and descriptive adequacy apart from issues in MT theory. Many other phenomena not described in those treatments (since they are unlikely to occur in human translation) are captured;  trivial but ubiquitous examples are character conversion errors and punctuation in numerals.
This study analyzes surface effects, rather than their causes (for example, an unknown word can cause a missed parse, which causes many surface problems to occur). We are assessing only surface problems without trying to reconstruct their original cause, for two important reasons:  (1) the actual cause of errors is unknown to those who are doing the classification; and, (2) from a user’s perspective, the causes are generally irrelevant to whether a system can produce output adequate for a particular task (assuming, as this test will, that user-adaptable features of a system have been optimized).
Preparing the Test Suite
The distribution of texts by their acceptability for particular text-handling tasks, as judged by users who routinely perform those tasks, will have clusters of translation  problems of particular types.  Many problems of many types will be in the set found to be acceptable for the most tolerant tasks (e.g., filtering).  Fewer problems of each type, and, more significantly, fewer types of problems, will appear in texts rated acceptable for the less tolerant tasks (e.g., extraction).  The direct identification by users of potentially diagnostic phenomena provides clues to the identification and weighting scheme of translation problems.   Ideally, the point on the text-handling task order where problems of a particular  type disappear from the acceptable texts means that errors of that type are diagnostic at that juncture.  Patterns representing such problem types are therefore represented in the ultimate MT Proficiency Scale test set.
After classifying translation problems into categories, and determining which problem categories appear to be diagnostic at which level on the text-handling task order, the actual MT Functional Proficiency Scale diagnostic test can be developed.  This process involves creating Japanese patterns which correspond to diagnostic types (or simply using exemplars from the corpus, where these display one problem category rather than many, as is often the case).  Many example patterns of many categories are developed into a simple text file, along with the lexicon for the patterns, so that systems with lexical development can train optimally for the MT Proficiency Scale test.
When run against any Japanese-English MT system, the output of the translation of this simple file will be scored by comparing the output of the patterns against expert translations of them.  Clearly, such scoring is a subjective assessment, since, as noted above, there is no one “right” translation. Minimizing the subjectivity of these judgments remains an issue.  However, the patterns will be designed so that a judgment about the fidelity and intelligibility of the translation of a pattern should be straightforward.
The scoring should reveal a cutoff between a pattern series for which the trained MT system did reasonably well (e.g., translated most of the patterns acceptably) and a series where it failed more often.  This will essentially indicate the level of proficiency, and can be related immediately to the highest point on the text handling tasks hierarchy for which the system output is useful
Conclusion
The exercises that are part of the MT Proficiency Scale project attempt to capture the practical realities of translation in the context of a certain task and user population. Each in the series of exercises has a purpose, and validation can be drawn from among the exercise results.  This study and the metric development that follows from it reveal important information about the way in which machine translation is used today, and how MT can be tuned to perform consistently for specific user tasks.
There are several critical issues to be confronted in the course of this development, in particular the merging of user judgments over many user groups and missions into a single hierarchical scale of text handling tasks, and the accurate representation of translation problems in the final diagnostic test set. These and other issues will require careful analysis of user judgments and translation phenomena.
Subsequent study will determine to what extent the processes, and perhaps even data, of the current project can be extended to other language pairs. Ultimately, the MT Proficiency Scale will serve as a reusable, potentially self-testing means of determining the performance of an MT system for the actual purposes to which MT will be applied now and in the future.
The overall methodology will provide a quick, inexpensive and portable diagnostic set to predict the suitability of an MT system's output for real uses in modern text handling and analysis.
References
Church, K. and Hovy, E. (1991).  Good Applications for Crummy Machine Translation.  In Jeannette G. Neal and Sharon M. Walter (eds.) Proceedings of the 1991 Natural Language Processing Systems Evaluation Workshop.  Rome Laboratory Final Technical Report RL-TR-91-362.
Connor-Linton, J. (1995).  Cross-cultural comparison of writing standards: American ESL and Japanese EFL.  World Englishes, 14.1:99-115. Oxford: Basil Blackwell.
Dostert, B. (1973).  User's Evaluation of Machine Translation, Georgetown MT System, 1963-1973. Rome Air Development Center Report AD-768 451.  Texas A&M University.
Pierce, J., Chair. (1966).  Language and Machines:  Computers in Translation and Linguistics.   Report by the Automatic Language Processing Advisory Committee (ALPAC).  Publication 1416.  National Academy of Sciences National Research Council.
Van Slype, G. (1979).  Critical Methods for Evaluating the Quality of Machine Translation. Prepared for the European Commission Directorate General Scientific and Technical Information and Information Management.  Report BR 19142.  Bureau Marcel van Dijk.
White, J., and O'Connell, T.A. (1994).  The ARPA MT evaluation methodologies: evolution, lessons, and future approaches.  Proceedings of the 1994 Conference, Association for Machine Translation in the Americas.
White, J., and O’ Connell, T. A. (1996).  Adaptation of the DARPA machine translation evaluation paradigm to end-to-end systems.  Proceedings of AMTA-96.
White, J. S.,  and Taylor, K. B. (1998).  A Task-Oriented Evaluation Metric for Machine Translation.  Proceedings of LREC-98, Volume I. 21-27.