We propose an error analysis framework for automatic summarization. The framework presented herein incorporates five problems that cause automatic summarization systems to produce errors and three metrics for quality. We classify errors in automatic summaries into 15 categories comprising a combination of the three quality metrics and five problems. We also present a method to classify automatic-summary errors into these categories. Using our error analysis framework, we analyze the errors in an automatic summary produced by our system and present the results. We use these results to refine our system and then show that the quality of the automatic summary is improved. The error analysis framework that we propose is demonstrably useful for improving the quality of an automatic summarization system.

View full abstract

This paper reports error analysis results on the product attribute value extraction task. We built the system that extracted attribute values from product descriptions by simply matching the descriptions and entries in an attribute value dictionary. The dictionary is automatically constructed by parsing semi-structured data such as tables and itemizations in product descriptions. We run the extraction system on the corpus where product attribute values were annotated by a single subject, and then investigated false-positives and false-negatives. We conducted the error analysis procedure on 100 product pages extracted from five different product categories of an actual e-commerce site, and designed error type categories according to the results of the error analysis on those product pages. In addition to show the error type categories and their instances, we also discuss processing and data resources required for reducing the number of error instances.

View full abstract

In general, there are two types of dialogue systems: the task-oriented dialogue system and the non-task-oriented or chat dialogue system. In recent years, chat dialogue systems have received much attention mainly because of the advances in automatic knowledge acquisition from the web. Nevertheless, few studies are dedicated to the error analysis of chat dialogue systems. This is in contrast with the many error-analysis-related studies on task-oriented dialogue systems. An error in a chat dialogue system can lead to the dialogue breakdown, where users are no longer willing to continue the conversation. Therefore, error analysis is crucial in such systems. However, it is difficult to analyze errors in chat dialogue systems because of the complex internal structures of the systems. In the present study, we analyze and categorize the errors in a text chat dialogue system on the basis of the surface form of the conversations. We construct a chat dialogue corpus between a chat system and users and analyze the dialogue breakdowns included in the corpus.

View full abstract

Error analysis is used to improve accuracy of machine translation (MT) systems. Various methods of analyzing MT errors have been proposed; however, most of these methods are based on differences between translations and references that are translated independently by human translators, and few methods have been proposed for manual error analysis. This work proposes a method that uses a machine learning framework to identify errors in MT output, and improves efficiency of manual error analysis. Our method builds models that classify low and high quality translations, then identifies features of low quality translations to improve efficiency of the manual analysis. Experiments showed that by using our methods, we could improve the efficiency of MT error analysis.

View full abstract

The Todai Robot Project aims at integrating various AI technologies including natural language processing (NLP), as well as uncovering novel AI problems that have been missed while the fragmentation of the research field, through the development of software systems that solve university entrance exam problems. Being primarily designed for the measurement of human intellectual abilities, university entrance exam problems serve as an ideal benchmark for AI technologies. They also enable a quantitative comparison between the AI systems and human test takers. This paper analyzes the errors made by the software systems on the mock university entrance exams hosted by a popular preparatory school. Based on the analyses, key problems towards higher system performances and the current issues in the field of NLP are discussed.

View full abstract