A Study on Building a Japanese General-Purpose Language Model for Temporal Common Sense Understanding

Abstract

In order to understand events expressed in natural language, it is important to understand time. However, since they are often omitted from descriptions, it is necessary to have common sense knowledge about various temporal aspects of events. Therefore, we aim to build a Japanese general-purpose language model that can identify time-related common sense using the English dataset on temporal common sense, MC-TACO, translated into Japanese. We have conducted experiments with several masking rates and settings: (i) masking random tokens, (ii) masking words of the answer part in the input (iii) masking temporal-related words, (iv) masking words with high attention score, and (v) masking words with high saliency score. Through our experiments, we have confirmed that high accuracy can be obtained in Japanese with the same settings as in English.