This paper describes our dataset of Japanese cloze questions designed for the evaluation of machine reading comprehension. The dataset consists of questions automatically generated from Aozora Bunko, and each question is defined as a 4-tuple: a context passage, a query holding a slot, an answer character, and a set of possible answer characters. The query is generated from the original sentence, which appears immediately after the context passage on the target book, by replacing the answer character with the slot. The set of possible answer characters consists of the answer character and the other characters who appear in the context passage. Because the context passage and the query share the same context, a machine that precisely understands the context may select the correct answer from the set of possible answer characters. The unique point of our approach is that we focus on characters of target books as slots to generate queries from original sentences because they play important roles in narrative texts and a precise understanding of their relationship is necessary for reading comprehension. To extract characters from target books, manually created dictionaries of characters are employed because some characters appear as common nouns, not as named entities.

To achieve deep-level learning by students in interactive teaching through questions, it is necessary to analyze students’ answers and evaluate them in detail. Automatic Short Answer Grading (ASAG) is a technique for grading short sentences such as students’ answers. In recent years, ASAG has achieved high accuracy by using large-scale language models. However, existing methods tend to rely on the word agreement with the model answer and thus cannot provide detailed evaluation for feedback. In this study, we propose an evaluation method combining the knowledge graph of the textbook and the pre-trained language model to evaluate students’ answers in detail．We constructed a knowledge graph to represent the knowledge in the textbook systematically. When constructing a knowledge graph from a textbook, there are two possible methods: one is to construct a knowledge graph based on the relationships among words in the textbook, and the other is to construct a knowledge graph by retaining the syntactic information of the sentences in the textbook. Therefore, we constructed knowledge graphs using these two methods and verified which knowledge graph was more effective. In addition, the existing ASAG datasets do not include a detailed evaluation of the answers. Therefore, we collected questions and sample answers related to Informatics I, a high school course, and constructed a dataset of Informatics I questions with seven evaluation labels. Using the constructed knowledge graph and the Informatics I question dataset, we verified the effectiveness of the evaluation method for answers based on the knowledge graph in the textbook. As a result, we found that the knowledge graph based on word relationships fluctuates the structural information of answers. On the other hand, knowledge graphs that retain syntactic information of sentences do not change the syntactic information of answers and provide effective information for evaluation, and the Knowledge-aware Answer Grading Model from K-BERT was found to be an effective method for evaluation based on knowledge graphs.

Machine reading comprehension methods that generate answers by referring to multiple passages for a question have gained much attention in AI and NLP communities. The current methods, however, do not investigate the relationships among multiple passages in the answer generation process, even though topics correlated among the passages may be answer candidates. Our method, called neural answer Generation through Unified Memories over Multiple Passages (GUM-MP), solves this problem as follows. First, it determines which tokens in the passages are matched to the question. In particular, it investigates matches between tokens in positive passages, which are assigned to the question, and those in negative passages, which are not related to the question. Next, it determines which tokens in the passage are matched to other passages assigned to the same question and at the same time it investigates the topics in which they are matched. Finally, it encodes the token sequences with the above two matching results into unified memories in the passage encoders and learns the answer sequence by using an encoder-decoder with a multiple-pointer-generator mechanism. As a result, GUM-MP can generate answers by pointing to important tokens present across passages. Evaluations indicate that GUM-MP generates much more accurate results than the current models do.