Controllable Japanese Temporal Inference Dataset

Abstract

Natural Language Inference (NLI) tasks that require temporal inference remain challenging for pre-trained language models (LMs). Although various datasets have been created for this task, they primarily focus on English and do not address the need for resources in other languages. In this paper, we present a Japanese NLI benchmark for temporal inference. To begin the data annotation process, we create inference templates consisting of various inference patterns based on the formal semantics test suites. We then automatically generate diverse NLI examples by assigning nouns, verbs, and temporal expressions to the templates using the Japanese case frame dictionary. We evaluate the generalization capacities of monolingual/multilingual LMs by using controlled splits of our dataset. Our findings demonstrate that LMs struggle with handling specific linguistic phenomena such as habituality.