Construction of a Persona Dialogue Dataset Collected from a Self-publishing Novel Website

Abstract

In recent years, research on character role-playing using large language models (LLMs) has been actively conducted. One approach for evaluating role-playing ability is to verify whether an LLM can respond in line with a given persona, which consists of information about the character. In order to accurately assess the role-playing ability, it is important to have the LLM perform role-playing for characters it has not been trained on before. However, many of the datasets for role-playing tasks proposed so far contain characters from well-known works, which may have appeared frequently in the LLM's pre-training data. As a result, there is a risk that the LLM's ability to utilize the persona may not be evaluated accurately. To address this, we have constructed a persona-based dialogue dataset by collecting dialogue from 608 characters across 96 online novels, including lesser-known works. The experimental results show that fine-tuning is important for improving the role-playing ability of LLMs using personas. On the other hand, we found challenges in the generalization performance of role-playing abilities for characters not included in the training data. This suggests that the dataset could be useful for exploring learning methods to improve generalization performance.