Development of Multimodal Dialogue Platform Balancing Reactive Behavior and Response Utterances from Multimodal Inputs

Abstract

In real-world spoken dialogues, it's crucial not only to exchange verbal information but also to accurately reflect the non-verbal behaviors of both users and the system within the interaction. However, the exchange of non-verbal information in human-to-human conversations is processed at high speeds, on the order of tens of milliseconds, making it challenging to balance accuracy and speed within a single dialogue control model. Moreover, the non-verbal information necessary varies with the dialogue task, and it is desirable to be able to switch sensors, recognition models, output devices, etc., depending on the context. Therefore, we have been developing a multimodal dialogue platform, utilizing the distributed processing structure of ROS2, that allows for the parallel, distributed execution of various multimodal input recognizers, dialogue models, and output control modules. In this session, we will provide a simple dialogue demonstration for participants to experience, as well as explain the platform we are developing.