Do Large Language Models Understand Japanese Honorifics Based on Contextual Information?

Abstract

Using honorifics properly is essential for maintaining harmonious relationships with others when communicating in Japanese. Japanese Honorifics have both grammatical aspects (e.g., verb conjugation) and contextual ones (e.g., social relationships among people). Therefore, a precise understanding of honorifics is a challenging task for systems because it requires knowledge of grammatical rules and the ability to understand contexts. While large language models are known to perform well in Japanese tasks, no existing dataset has aimed to evaluate these models’ performance of using Japanese honorifics flexibly according to contextual information. In this thesis, we introduce two honorific understanding tasks that require contextual information: an acceptability judgment task regarding the usage of honorifics and an honorific conversion task. We first construct a new Japanese honorifics dataset using a template-based method to generate data in a controlled way. We also sample the data from an existing Japanese honorifics corpus and annotate them with the additional information to evaluate the models with more natural data. Using our datasets, we then conduct experiments to evaluate the performance of large language models, including GPT-4, on the two tasks from multiple perspectives. Our experimental results demonstrated that the models still have room for improvement in sentences with complicated structures compared to the simpler ones in the honorific conversion task.