The task of detecting words with semantic differences across corpora is primarily addressed by word representations such as Word2Vec or BERT. However, there are no abundant computing resources available in the real world where linguists and sociologists apply these techniques. In this paper, we extend an existing CPU-trainable model which trains vectors of all time periods simultaneously. Experimental results demonstrate that the extended models achieved comparable or superior results to strong baselines in English corpora, SemEval-2020 Task 1, and Japanese. Furthermore, we compared the training time of each model and conducted a comprehensive analysis of Japanese corpora.

Zero-shot relation extraction aims to recognize (new) unseen relations that cannot be observed during training. Due to this point, recognizing unseen relations with no corresponding labeled training instances is a challenging task. Recognizing an unseen relation between two entities in an input instance at the testing time, a model needs to grasp the semantic relationship between the instance and all unseen relations to make a prediction. This study argues that enhancing the semantic correlation between instances and relations is key to effectively solving the zero-shot relation extraction task. A new model entirely devoted to this goal through three main aspects was proposed: learning effective relation representation, designing purposeful mini-batches, and binding two-way semantic consistency. Experimental results on two benchmark datasets demonstrate that our approach significantly improves task performance and achieves state-of-the-art results. Our source code and data are publicly available.

This study constructed an error-tagged evaluation corpus for Japanese grammatical error correction (GEC). Evaluation corpora are essential for assessing the performance of models. The availability of various evaluation corpora for English GEC has facilitated a comprehensive comparison between models and the development of the English GEC community. However, the development of the Japanese GEC community has been hindered due to the lack of available evaluation corpora in the Japanese GEC. As a result, we constructed a new evaluation corpus for the Japanese GEC and made it available to the public. We used texts written by the Japanese language learners in the Lang-8 corpus, a representative learner corpus in GEC, to create the evaluation corpus. The specification of the evaluation corpus was modified to align with the representative corpora and tools in the English GEC, making it easy for GEC researchers and developers to use the evaluation corpus. Finally, we evaluated representative GEC models on the created evaluation corpus and reported baseline scores for future Japanese GEC.

Character-based word segmentation models have been extensively applied to Asian languages, including Thai, owing to their promising performance. These models estimate the word boundaries from a character sequence; however, a Thai character unit in a sequence has no inherent meaning, in contrast with word, subword, and character cluster units that represent more meaningful linguistic information. In this paper, we propose a Thai word segmentation model that uses various types of information, including words, subwords, and character clusters, from a character sequence. Our model applies multiple attentions to refine segmentation inferences by estimating the significant relationships among characters and various unit types. We evaluated our model on three Thai datasets, and the experimental results show that our model outperforms other Thai word segmentation models, demonstrating the validity of using character clusters over subword units. A case study on sample Thai text supported these results. Thus, according to our analysis, particularly the case study, our model can segment Thai text accurately, while other existing models yield incorrect results that violate the Thai writing system.

Encoder-decoder models have been commonly used; they have achieved state-of-the-art results for many natural language generation tasks. However, according to the reports of previous studies, encoder-decoder models suffer from generating redundant repetitions. Thus, we herein propose a repetition reduction module (RRM) for encoder-decoder models that estimates the semantic difference of a source sentence before and after it is fed into the model to capture the consistency between the two sides. As an autoencoder, the proposed mechanism supervises the training of encoder-decoder models to reduce the number of repeatedly generated tokens. The evaluation results of the publicly available machine translation and response generation datasets demonstrate the effectiveness of our proposal.

Natural language inference is a core natural language understanding task for determining whether a hypothesis is true (entailment), false (contradiction), or neither (neutral) when a set of premises is true. Logical entailment and implicature can differ when an inference contains numeral expressions. Embedding numeral expressions in contexts such as negation and conditionals can enable reversing the entailment relation between a premise and a hypothesis to that embedded in general contexts. Furthermore, numeral expressions in Japanese are characterized by the flexibility of quantifier positions, the variety of numeral suffixes, and their usages. However, studies on developing annotated corpora focusing on these features and benchmark datasets for understanding Japanese numeral expressions have been limited. In this study, we developed a corpus that annotates each numeral expression in an existing phrase-structure-based Japanese treebank with its numeral suffix, position, and usage types. Furthermore, we constructed an inference dataset for numerical expressions based on our annotated corpus. Our inference dataset focused on inferences in which their entailment labels differ between logical entailment and implicature and contexts such as negations and conditionals where entailment labels can be reversed. Experiments were conducted using the constructed inference dataset to investigate the extent to which current standard pre-trained language models can handle inferences that require an understanding of numeral expressions. The results confirmed that the Japanese bidirectional encoder representation model cannot satisfactorily handle inferences involving various numeral expressions.

A character sequence comprises at least one or more segmentation alternatives. This can be considered segmentation ambiguity and may weaken segmentation performance in word segmentation. Proper handling of such ambiguity lessens ambiguous decisions on word boundaries. Previous works have achieved remarkable segmentation performance and alleviated the ambiguity problem by incorporating the lattice, owing to its ability to capture segmentation alternatives, along with graph-based and pre-trained models. However, multiple granularity information, including character and word, in a lattice that encodes with such models may not be attentively exploited. To strengthen multi-granularity representations in a lattice, we propose the Lattice ATTentive Encoding (LATTE) method for character-based word segmentation. Our model employs the lattice structure to handle segmentation alternatives and utilizes graph neural networks along with an attention mechanism to attentively extract multi-granularity representation from the lattice for complementing character representations. Our experimental results demonstrated improvements in segmentation performance on the BCCWJ, CTB6, and BEST2010 datasets in three languages, particularly Japanese, Chinese, and Thai.

Neural models trained on large amounts of parallel data have achieved impressive performance in abstractive summarization tasks. However, constructing large-scale parallel corpora can be expensive and challenging. In this work, we introduce a low-cost and effective strategy called ExtraPhrase to augment training data for abstractive summarization tasks. ExtraPhrase constructs pseudo training data with two modules: sentence compression and paraphrasing. We extract major parts of an input text with sentence compression and obtain its diverse expressions with paraphrasing. Through experiments, we show that ExtraPhrase improves the performance of abstractive summarization tasks by more than 0.50 points in ROUGE scores compared to the setting without data augmentation. ExtraPhrase also outperforms existing methods such as back-translation and self-training. We also show that ExtraPhrase is significantly effective when the amount of genuine training data is remarkably small, i.e., in a low-resource setting. Moreover, ExtraPhrase is more cost-efficient than existing approaches.

Named entity recognition (NER) is one of the core technologies for knowledge acquisition from text and has been used for knowledge extraction of chemicals, medicine, and so on. As one of the NER improvement approaches, multi-task learning that learns a model from multiple training data has been used. Among multi-task learning, an auxiliary learning method, which uses training data of an auxiliary task for improving its target task, has shown higher NER performance than conventional multi-task learning for improving all the tasks simultaneously. The conventional auxiliary learning method uses only one auxiliary training dataset. We propose Multiple Utilization of NER Corpora Helpful for Auxiliary BLESsing (MUNCHABLES). MUNCHABLES utilizes multiple training datasets as auxiliary training data by the following methods: the first one is to fine-tune the NER model of the target task by sequentially performing auxiliary learning for each auxiliary training dataset, and the other is to use all training datasets in one auxiliary learning. We evaluate MUNCHABLES on eight chemical/biomedical/scientific domain NER tasks, where seven training datasets are used as auxiliary training data. The experimental results show that our methods achieve higher micro and macro average F1 scores than a conventional auxiliary learning method using one auxiliary training dataset and conventional multi-task learning method. Furthermore, our method achieves the highest F1 score on the s800 dataset.

Because open-domain dialogues allow diverse responses, common reference-based metrics for text generation, such as bleu, do not correlate well with human judgments unless we prepare an extensive reference set of high-quality responses for input utterances. In this study, we propose a fully automatic, uncertainty-aware evaluation method for open-domain dialogue systems, υbleu. Our method first collects diverse reference responses from massive dialogue data, annotates their quality judgments by using a neural network trained on automatically collected training data, and then computes weighted bleu using the automatically-retrieved and -rated reference responses. We also employ this method with an embedding-based metric, berts_core, instead of the word-overlap-based metric, bleu, to absorb surface variations of the reference responses. The experimental results on the meta-evaluation of our evaluation method for dialogue systems based on massive Twitter data confirmed that our method substantially improves correlations between bleu (or berts_core) and human judgments. We also confirmed that our method is effective when it is combined with a reference-free metric.

This paper proposes a sequential edge-editing model for the document-level relation extraction that edits edges in a relation graph, where nodes are entities and edges are the output relations as candidates, to consider the interaction among relations. Deep learning models have been mainstream in recent document-level relation extraction research. However, adding new features to the model is difficult because there is no straightforward method for combining multiple models, and their implementation is different. To tackle this, we introduce the task of editing the relation candidates extracted by existing methods to take into account the interactions among relations. For the material synthesis procedure corpus, our proposed model revised the output of the rule-based extractor from 80.5% to 86.6% on F-scores, where a sequential edge-editing model performed 79.4% when extracting from scratch. On the other hand, performance did not improve for the MATRES corpus, the standard benchmark for temporal relation extraction, when the model edited the output of the state-of-the-art deep learning models. Analysis of these differences revealed that the significant factor contributing to the performance improvement is the difference between the extractable relations in the editing model from scratch and the relations before editing.

Gathering citizen feedback, analyzing it, and comparing the results against other cities is essential for improving government policy and service quality. However, because different cities have different policies and services, the opinions of citizens in different cities also differ. This makes it difficult to analyze citizen opinions adapted for multiple cities using machine learning. In this study, we propose a method for extracting citizen opinions across cities. We evaluated our proposed method using a tweet dataset collected from citizens of Yokohama, Sapporo, and Sendai, confirming its effectiveness to fine-tune a model using the source city and re-fine-tune it with a few tweets from the target city. We clarified that training data in the target city can be effectively selected using the model trained with tweets from the source city, with high confidence in the prediction.

Natural language inference (NLI) is the task of identifying the inferential relation between a text pair. In recent times, neural-based approaches have achieved high performance in NLI. However, they are unable to explain their reasoning processes. On the other hand, symbolic approaches have the advantage that their reasoning process is understandable to humans. This paper proposes a method for integrating a neural NLI model and the tableau proof system, with the latter explaining the reasoning processes. The standard tableau method decomposes logical formulas by applying inferential rules and checks for a valuation that satisfies the given constraints. Unlike the standard tableau method, the proposed method uses dependency structures as its components rather than logical formulas and employs a neural NLI model for the latter process. To analyze the behavior of our method, we conducted an experiment on the neural NLI model and the proposed method using SNLI corpus. In addition, we formalize our method model-theoretically and clarify the theoretical limitations of this method based on the formalization.

We conducted a study to determine what kind of structural knowledge learned in neural network encoders is transferable to the processing of natural language. We designed artificial languages with structural properties that mimic those of natural language, pretrained encoders on the data, and examined the encoders' effects on downstream tasks in natural language. Our experimental results demonstrate the importance of statistical dependency, as well as the effectiveness of the nesting structure in implicit dependency relations. These results indicate that position-aware context dependence represents knowledge transferable across different languages.

In this paper, we explore the capacity of a language model-based method for grammatical error detection in detail. We first show that 5 to 10% of training data are enough for a BERT-based error detection method to achieve performance equivalent to what a non-language model-based method can achieve with the full training data; recall improves much faster with respect to training data size in the BERT-based method than in the non-language model method. This suggests that (i) the BERT-based method should have a good knowledge of the grammar required to recognize certain types of error and that (ii) it can use the knowledge to estimate whether the given word is correct or erroneous after fine-tuning with few training samples, which explains its high generalization ability in grammatical error detection. We further show with pseudo error data that it actually exhibits such nice properties for recognizing various types of error. Finally, based on these findings, we discuss a cost-effective method for detecting grammatical errors with feedback comments to learners.

The meaning and usage of words change over time. One method of analyzing these changes is to group word tokens by their meanings in each period and compare their usage rates. Several methods of this kind have been used to analyze semantic changes in English, but they have not yet been applied to Japanese. In addition, the methods have not been compared. Therefore, the performance of this method on Japanese and the conditions under which each method is effective have not been clarified. Thus, we conducted the following experiments on Japanese words. We applied a supervised grouping method using a dictionary and an unsupervised grouping method using clustering to context-dependent vectors in the BERT model and compared them. We also pre-trained BERT on a diachronic corpus and analyzed the diachronic features captured by the context-dependent vectors in BERT. The results of the comparison and analysis showed that in the absence of a well-developed dictionary, the clustering-based method was better able to capture semantic change. Furthermore, it was found that fine-tuning with a diachronic corpus can be used to capture semantic changes in older periods. However, it was also found that some words with usages that did not appear in the older period could not always be captured.

In this paper, I report on my experience supporting the production of a translated list of participants in the 2020 Tokyo Olympic Games using machine translation. Producing translated lists of the participants in Olympic Games events is difficult owing to the size and complexity of the games and the wide variety of participating countries. To address this challenge, I implemented a system to translate the names of specific persons called Tsuduri2021, which consists of a translation dictionary and 208 country-specific transliteration subsystems. Each of these subsystems comprises a transducer operating based on different country-specific models produced by Awase2019. The results of a comparison between the final translated list of participants and the output of the Tsuduri2021 system show that most of the computed results were actually used in the final list, including 90.4% of the full name translations and 94.0% of the single-name translations.

Interviews are an efficient way to elicit knowledge from experts in different domains. In this paper, we introduce a corpus of interview dialogues on subjects in culinary arts called CIDC. In collecting this data, interviewers played an active role in eliciting culinary knowledge from cooking experts. The corpus consists of 308 interview dialogues, each about 13 minutes long, comprising a total of 64,000 utterances. We used a videoconferencing tool to collect the data, which included the content that the participants shared on their screens. This approach allowed us to classify the facial expressions of the interlocutors from the video data. We also categorized the experts as either professionals or enthusiasts and the interviewers as skilled or unskilled. We expect this corpus to facilitate future research on knowledge elicitation mechanisms in interview dialogues.

Named-entity recognition (NER) technologies have been developed for the general knowledge domain, and the extension of NER to specific domains has now become a topic of active research. In this paper, we provide a survey of recent studies on domain-specific NER technologies for the Japanese language. We focus on aspects of different applications such as their targets and purposes as well as the knowledge domains considered. Our results show that chemistry is the most fundamental knowledge domain represented in these works, along with medical, financial, machining, literature, and food domains. Most of the systems included in the survey utilized machine learning methods such as BiLSTM-CRF and BERT, though some studies used rules and dictionaries in addition to these approaches.