Collection of Multimodal Dialogue Data Using Heterogeneous Sensors and Analysis of the Relationship between Sensor Information and Subjective Evaluation

Abstract

In order to accurately understand user state and respond smoothly to it, dialogue systems are expected to utilize users' multimodal information obtained from heterogeneous sensors, in addition to information obtained from speech alone. Therefore, in this study, we collected dialogue data in an attempt to implement a dialogue system that utilizes users' multimodal information obtained from heterogeneous sensors. Specifically, we used heterogeneous sensors to collect data such as speech, video, physiological signals, gaze information, and body movement information during a dialogue. Additionally, after each dialogue, the speaker conducted a subjective evaluation of their own mental states while reviewing the video of the dialogue. This paper presents the results of dialogue data collection and the analysis of the relationship between sensor information and subjective evaluations.