Automatic Short Answer Grading to Pedagogical Questions Using Knowledge Graphs and Pre-trained Models

Abstract

To achieve deep-level learning by students in interactive teaching through questions, it is necessary to analyze students’ answers and evaluate them in detail. Automatic Short Answer Grading (ASAG) is a technique for grading short sentences such as students’ answers. In recent years, ASAG has achieved high accuracy by using large-scale language models. However, existing methods tend to rely on the word agreement with the model answer and thus cannot provide detailed evaluation for feedback. In this study, we propose an evaluation method combining the knowledge graph of the textbook and the pre-trained language model to evaluate students’ answers in detail．We constructed a knowledge graph to represent the knowledge in the textbook systematically. When constructing a knowledge graph from a textbook, there are two possible methods: one is to construct a knowledge graph based on the relationships among words in the textbook, and the other is to construct a knowledge graph by retaining the syntactic information of the sentences in the textbook. Therefore, we constructed knowledge graphs using these two methods and verified which knowledge graph was more effective. In addition, the existing ASAG datasets do not include a detailed evaluation of the answers. Therefore, we collected questions and sample answers related to Informatics I, a high school course, and constructed a dataset of Informatics I questions with seven evaluation labels. Using the constructed knowledge graph and the Informatics I question dataset, we verified the effectiveness of the evaluation method for answers based on the knowledge graph in the textbook. As a result, we found that the knowledge graph based on word relationships fluctuates the structural information of answers. On the other hand, knowledge graphs that retain syntactic information of sentences do not change the syntactic information of answers and provide effective information for evaluation, and the Knowledge-aware Answer Grading Model from K-BERT was found to be an effective method for evaluation based on knowledge graphs.