A spoken dialogue system that listens to complaints and supports decision-making based on the speaker's emotions

Abstract

In this report, we discuss the design of prompts for a large language model in a spoken dialogue system that listens to complaints and supports decision-making. The system explicitly divides the dialogue into three phases: listening to complaints, supporting decision-making, and casual conversation. Information relevant to each phase is structured into slots, ensuring consistency in system behavior and facilitating appropriate phase transitions. Moreover, the system continuously estimates the underlying emotions behind the speaker's utterances, rather than merely capturing their surface meaning, and generates responses accordingly. This enhances the naturalness of empathetic and listening behaviors, such as expressing sympathy and asking relevant questions. Additionally, to improve the coherence of system behavior, we introduce both general considerations that apply across all phases and phase-specific guidelines, reducing subtle unnaturalness and discomfort. We demonstrate the effectiveness of this design by presenting actual prompts and dialogue examples.