Automatic evaluation of Machine Translation (MT) quality is essential to develop high-quality MT systems. Various evaluation metrics have proposed, and among them, BLEU is widely used as the de facto standard metric. BLEU counts N-grams common between reference and hypothesis translation. On the other hand, ROUGE-L counts longest common subsequences. However, these methods have some problems. People give high scores to Rule-based MT (RBMT), but these methods do not, because RBMT tends to use alternative words. Conventional metrics are severe against the difference of words, but people accept them if the translation has the same meaning. Statistical MT (SMT) tends to translate “A because B” as “B because A” in case of translation between Japanese and English. BLEU does not care about global word order, and this severe mistake is not penalized very much. In order to consider global word order, this paper proposes a lenient automatic evaluation metric based on rank correlation of word order. By focusing on only words common between the two translations, this method is lenient with the use of alternative words. The difference of words is measured by precision of words, and its weight is controlled by a parameter. By using submissions of NTCIR-7 & 9’s Patent Translation task, the proposed method outperforms conventional measures in terms of system level comparison.

View full abstract

Recent developments in computer technology have allowed the construction and widespread application of large-scale speech corpora. To enable users of speech corpora to easier data retrieval, we attempt to characterise the speaking style of speakers recorded in the corpora. We first introduce the three scales for measuring speaking style which were proposed by Eskenazi in 1993. We then use morphological features extracted from speech transcriptions that have proven effective in discriminating between styles and identifying authors in the field of natural language processing to construct an estimation model of speaking style. More specifically, we randomly choose transcriptions from various speech corpora as text stimuli with which to conduct a rating experiment on speaking style perception. Then, using the features extracted from these stimuli and rating results, we construct an estimation model of speaking style, using a multi-regression analysis. After cross-validation (leave-1-out), the results show that among the three scales of speaking style, the ratings of two scales can be estimated with high accuracy, which proves the effectiveness of our method in the estimation of speaking style.

View full abstract

We formalized and implemented a method for solving comprehension questions referring to underlined passages in Contemporary Japanese of the National Center Test. A target question consists of a text body extracted from a critical essay or novel, a question sentence, and five choices; the question sentence refers to an underlined passage in the text body and asks a question related to the passage, such as the interpretation of the passage, the reason for the author’s claim (in the essay), and the emotional state or feeling of a character (in the novel). Given a question, the method determines which text segment is the key to the question and selects the choice that is most similar to the segment. The method correctly solved more than half the “critical essay” questions in previous tests of the National Center Test.

View full abstract

In statistical machine translation, Chinese and Japanese is a well-known long-distance language pair that causes difficulties to word alignment techniques. Pre-reordering methods have been proven efficient and effective; however, they need reliable parsers to extract the syntactic structure of the source sentences. On one hand, we propose a framework in which only part-of-speech (POS) tags and unlabeled dependency parse trees are used to minimize the influence of parse errors, and linguistic knowledge on structural difference is encoded in the form of reordering rules. We show significant improvements in translation quality of sentences in the news domain over state-of-the-art reordering methods. On the other hand, we explore the relationship between dependency parsing and our pre-reordering method from two aspects: POS tags and dependencies. We observe the effects of different parse errors on reordering performance by combining empirical and descriptive approaches. In the empirical approach, we quantify the distribution of general parse errors along with reordering quality. In the descriptive approach, we extract seven influential error patterns and examine their correlations with reordering errors.

View full abstract

Picture books have a significant influence on children’s language development. However, the sentences in picture books are difficult to analyze automatically. Therefore, to improve the accuracy of the morphological analysis of such sentences, we propose an automatic method to transform existing resources into applicable training data for picture books. In this paper, we first compare picture books with common corpora and then analyze the reasons for the difficulty in morphological analysis. Based on this analysis, we propose a transforming method for existing resources and show its effectiveness using the learning function of an existing morphological analyzer. Second, we perform further experiments using annotated data of picture books themselves. Then we reveal that our proposed method provides us with the same effect, with around 11,000 lines, that is 90,000 morphological annotations of picture books. In addition, we demonstrate an effective annotation strategy by investigating the learning curves and change in error types. In a discussion, we analyze the results focused on a picture book’s target ages and difficult to learn words and then further refine our proposed method. Finally, we also briefly consider the applicability of our method to other domains.

View full abstract

We propose a method of collective sentiment classification that assumes dependencies among labels of an input set of reviews. The key observation behind our method is that the distribution of polarity labels over reviews written by each user or written on each product is often skewed in the real world; intolerant users tend to report complaints while popular products are likely to receive praise. We encode these characteristics of users and products (referred to as user leniency and product popularity) by introducing global features in supervised learning. To resolve dependencies among labels of a given set of reviews, we explore two approximated decoding algorithms, “easiest-first decoding” and “two-stage decoding.” Experimental results on real-world datasets with user and/or product information confirm that our method contributed greatly to classification accuracy.

View full abstract

In Japanese, zero references often occur and many of them are categorized into zero exophora, in which a referent is not mentioned in the document. However, previous studies have focused only on zero endophora, in which a referent explicitly appears. We present a zero reference resolution model considering zero exophora and authors/readers of a document. To deal with zero exophora, our model adds pseudo-entities corresponding to zero exophora to candidate referents of zero pronouns. In addition, the model automatically detects mentions that refer to the author and reader of a document using lexico-syntactic patterns. We present particular behavior of authors/readers in a discourse as a feature vector of a machine learning model. The experimental results demonstrate the effectiveness of our model for not only zero exophora but also zero endophora.

View full abstract