Recently, as a replacement of syntactic tree-based approaches, such as tree-trimming, Long Short-Term Memory (LSTM)-based methods have been commonly used to compress sentences because LSTM can generate fluent compressed sentences. However, the performance of these methods degrades significantly while compressing long sentences because they do not explicitly handle long-distance dependencies between the words. To solve this problem, we proposed a higher-order syntactic attention network (HiSAN) that can handle higher-order dependency features as an attention distribution on LSTM hidden states. Furthermore, to avoid the influence of incorrect parse results, we trained HiSAN by maximizing the probability of a correct output together with the attention distribution. Experiments on the Google sentence compression dataset show that our method improved the performance from baselines in terms of F₁ as well as ROUGE-1, -2, and -L scores. In subjective evaluations, HiSAN outperformed baseline methods in both readability and informativeness. Besides, in this study, we additionally investigated the performance of HiSAN after training it without any syntactic dependency tree information. The results of our investigation show that HiSAN can compress sentences without relying on any syntactic dependency information while maintaining accurate compression rates, and also shows the effectiveness of syntactic dependency information in compressing long sentences with higher F₁ scores.

In recent years the amount of information on the Internet has increased exponentially．Consequently, automatic article summarisation technology will be indispensable．In this study, we propose a data augmentation method for an automatic summarisation system．The proposed method removes the least important sentence in an article．We used a topic model to determine the importance of sentences in articles. The Luhn and LexRank methods were used as comparative methods for determining the importance of sentences in articles. Additionally, we used Easy Data Augmentation (EDA) techniques as the comparison method for this study. EDA is a data augmentation method applied to document classification．A comparative experiment was performed using input datasets with 28,000, 57,000, and 287,226 articles．The Luhn and LexRank methods always produced the worst results, while EDA sometimes performed worse than the baseline method without a data augmentation. The proposed method performed the best in all cases．

Most machine translation (MT) research has focused on sentences as translation units (sentence-level MT), and has achieved acceptable translation quality for sentences where cross-sentential context is not required in mainly high-resourced languages. Recently, many researchers have worked on MT models that can consider a cross-sentential context. These models are often called context-aware MT or document-level MT models. Document-level MT is difficult to 1) train with a small amount of document-level data; and 2) evaluate, as the main methods and datasets focus on sentence-level evaluation. To address the first issue, we present a Japanese–English conversation corpus in which the cross-sentential context is available. As for the second issue, we manually identify the main areas where sentence-level MT fails to produce adequate translations in the lack of context. We then create an evaluation set in which these phenomena are annotated to alleviate the automatic evaluation of document-level systems. We train MT models using our corpus to demonstrate how the use of context leads to improvements.

The development of a reliable automatic evaluation metric of grammatical error correction (GEC) is useful for the research and development of GEC. Since it is difficult to cover all possible reference sentences, previous studies have proposed reference-less metrics. One of them achieved a higher correlation with manual evaluation than reference-based metrics by integrating metrics from the three perspectives of grammaticality, fluency, and meaning preservation. However, the correlation with the manual evaluation can be further improved because they are not considered for optimizing each metric for each manual evaluation. Therefore, in this study, we propose a method of optimizing each metric. Furthermore, we create a dataset with manual evaluation of system output that is ideal for optimization. Experimental results show that the proposed method improves correlation with the manual evaluation in both the metric of each perspective and combining the metrics. We also demonstrate that both using pre-trained language models for optimization and optimizing to manual evaluation on system output of GEC contribute to improvement. As a result of the analysis, it was found that our proposed metric appropriately rewarded more edits of error types than the conventional methods.

In this study, methods are proposed for generating diverse corrected sentences in grammatical error correction (GEC). There may be more than one way to correct a sentence that contains a grammatical error. However, existing GEC models do not consider the generation of diverse correction sentences. In a GEC task, all of the tokens are not rewritten in the text, but only those parts that need to be corrected. In this study, two methods are proposed for diversifying the output data in the GEC. First is a method to control the degree of correction by adding the information on it to the training data of the model, and second is a beam search method to diversify the output data considering the error points. The experimental results show that the existing methods are not suitable for diversification.

Neural Machine Translation (NMT) has shown drastic improvement in its quality when translating clean input such as text from the news domain. However, existing studies suggest that NMT still struggles with certain kinds of input with considerable noise, such as User-Generated Contents (UGC) on the Internet. To make better use of NMT for cross-cultural communication, one of the most promising directions is to develop a translation model that correctly handles these informal expressions. Though its importance has been recognized, it is still not clear as to what creates the large performance gap between the translation of clean input and that of UGC. To answer the question, we present a new dataset, PheMT, for evaluating robustness of MT systems against specific linguistic phenomena in Japanese-English translation. We provide more fine-grained error analysis about the behavior of the models with the accuracy and relative drop in translation quality on the contrastive dataset specifically designed for each phenomenon. Our experiments with the dataset revealed that not only our in-house models but even widely used off-the-shelf systems are greatly disturbed by the presence of certain phenomena.

In traditional NLP, we tokenize a sentence as a preprocessing, and thus the tokenization is unrelated to a downstream task. To address this issue, we propose a novel method to explore an appropriate tokenization for the downstream task. Our proposed method, Optimizing Tokenization (OpTok), is trained to assign a high probability to such appropriate tokenization based on the downstream task loss. OpTok can be used for any downstream task which uses a sentence vector representation such as text classification. Experimental results demonstrate that OpTok improves the performance of sentiment analysis, genre prediction, rating prediction, and textual entailment. The results also show that the proposed method is applicable to Chinese, Japanese, and English. In addition, we introduce OpTok into BERT, the state-of-the-art contextualized embeddings, and report a positive effect on the performance.

We present a word embedding-based monolingual phrase aligner. In monolingual phrase alignment, an aligner identifies the set of phrasal paraphrases in a sentence pair. Previous methods required large-scale lexica or high-quality parsers. Consequently, applying them to languages other than English is difficult. Unlike them, the proposed method uses only a pre-trained word embedding model, and thus it relies solely on raw monolingual corpora. Our method yields word alignments using pre-trained word embedding and then extends them to phrase alignments using a heuristic approach. Then, it composes a phrase representation from word embedding and searches for a set of consistent phrase alignments on a lattice of phrase alignment candidates. The experimental results in this study on the English dataset show that our method outperforms the previous phrase aligner. We also constructed a Japanese dataset for analysis, confirming that our method works with languages other than English.

We propose Implicit Quote Extractor, an end-to-end unsupervised extractive neural summarization model for conversational texts. When we reply to posts, quotes are used to highlight important parts of texts. We aim to extract quoted sentences as summaries. Most replies do not include quotes, so it is difficult to use quotes as supervision. However, even if it is not explicitly shown, replies always refer to certain parts of texts, and those parts can be presumed from the content of a reply. Those parts we call implicit quotes. Using replies, Implicit Quote Extractor aims to extract implicit quotes as summaries. The training task of the model is to predict whether a reply candidate is a true reply to a post. As a feature for prediction, the model has to choose a few sentences from the post. To predict accurately, the model adjusts the parameters to extract sentences that replies frequently refer to. We evaluate our model on two email datasets and one social media dataset, and confirm that our model is useful for extractive summarization. We further discuss two topics; one is whether quote extraction is an important factor for summarization, and the other is whether our model can capture salient sentences that conventional methods cannot.

This paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding regions of an image. The proposed visual attention mechanism captures the relationship between a word and an image region more precisely than a conventional visual attention mechanism trained through MNMT in an unsupervised manner. Our experiments on English-German and German-English translation tasks using the Multi30k dataset and on English-Japanese and Japanese-English translation tasks using the Flickr30k Entities JP dataset show that a Transformer-based MNMT model can be improved by incorporating our proposed supervised visual attention mechanism and that further improvements can be achieved by combining it with a supervised cross-lingual attention mechanism (up to +1.61 BLEU, +1.7 METEOR).

Understanding the influence of a training instance on a machine-learning model is important for interpreting the behavior of the model. However, it has been difficult and inefficient to evaluate the influence, considering how the prediction of a model would be changed if a training instance were not used. This prevents the application of influence estimation in neural networks with a large number of parameters. In this paper, we propose an efficient method for estimating the influence for neural networks. The method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. By switching between dropout masks, we can use sub-networks that learned or did not learn each training instance and estimate its influence based on the difference between the sub-networks. Through experiments with BERT and VGGNet on classification datasets, it was demonstrated that the proposed method can enhance the interpretability of error predictions. Quantitative evaluations were also performed by analyzing learning curves of sub-networks and applying the method to data filtering.

With a goal of building a dialogue system that can acquire food preference of users through conversation, this study proposes a method for selecting topics and generating questions based on a large-scale knowledge graph, Freebase. We define a topic as a relation between two entities in Freebase and create a topic embedding model that learns the similarity between topics based on the Wikipedia corpus. This model is used to select topics related to the current one. Moreover, we create a knowledge graph embedding, which is used to predict and complement missing entities in Freebase. These proposed methods enable to generate questions about user preferences while expanding the topic widely. We developed a web-based text chat system that generates questions based on the proposed methods and conducted a user study using crowd workers. Results demonstrate that the system can continue a dialogue longer by expanding the topics from a single dish. We also investigated the quality of the questions generated by the system. In addition, we showed that subjects received a better impression of the variation of topics and the continuity of contexts when the dialogue failure occurrence was below a certain level.

This paper proposes a new subword segmentation method for neural machine translation, called bilingual subword segmentation, which tokenizes sentences to minimize the difference between the number of subword units in a sentence and that in its translation. While existing methods tokenize a sentence without considering its translation, the proposed method tokenizes a sentence using subword units obtained from bilingual sentences and is thus suitable for machine translation. The method was evaluated on WAT Asian Scientific Paper Excerpt Corpus (ASPEC) English-to-Japanese, Japanese-to-English, English-to-Chinese, and Chinese-to-Ensglish translation tasks and WMT14 English-to-German and German-to-English translation tasks. The evaluation results reveal that the proposed method improves the performance of Transformer neural machine translation (up to +0.81 BLEU (%)).

Modeling the relations between text spans in a document is a crucial yet challenging problem for extractive summarization. Various kinds of relations exist among text spans of different granularity, such as discourse relations between elementary discourse units and coreference relations between phrase mentions. In this paper, we utilize heterogeneous graphs that contain multiple edge/node types to model the input document as well as the various relations among text spans in it. Also, we propose a heterogeneous graph based model for extractive summarization that considers the heterogeneity of the document graph. Experimental results on a benchmark summarization dataset verify the effectiveness of our proposed method.