An Evaluation Method for MT Systems and Its Application to ALT-J/E

Abstract

An evaluation method for a Japanese to English machine translation system which estimates translation quality based on translation technologies and source text characteristics is proposed. This method is applied to the translation of newspaper articles by the ALT-J/E system and further problems that need to be surmounted. Assuming 70% acceptability to be a practical level of quality, it is shown that morphological and dependency analysis technologies would require accuracy rate per word and per "bunsetsu" (which are Japanese phrase, in most case, consisting of a noun and "joshi" or post-positional word) as high as 99.8% and 99.4%, respectively. Current technology has reached this level. But the precision of translation for individual expressions is lower than expected values and the acceptability rate for the entire sentence drops to between 40% and 50%. To achieve an acceptability rate of 70%, it is necessary to achieve an average 7% improvement in 9 types of expressions and thus achieve a successful translation ratio of 96%. To upgrade the translation rate of these expressions, it will be necessary to establish techniques for meaning analysis of post-positional words, translation of expressions that combine noun clauses and compounded words, handling connections and embedded clauses, and verification of ellipses and paratactical constructions.