Generation of Surprise Expression in Dialogue by Using LLMs

Abstract

Expression of emotions is crucial in dialogue systems, and reaction of surprise is among them but not well explored. Surprise in dialogue can arise from various factors, dependent on knowledge and context, such as unexpected developments or the rarity of events. In this study, we evaluated three methods for generating surprise response in dialogue: (1) direct prediction using few-shot prompting with an LLM, (2) fine-tuning a BERT model on dialogue data, and (3) predicting the continuation of a dialogue with an LLM and judging surprise based on the discrepancy with the actual utterance. Our experiments demonstrated that the direct use of an LLM achieves the best performance. Further analysis of the reasoning behind GPT's judgments revealed instances where it incorrectly failed to exhibit surprise, even in surprising situations, citing reasons such as "it's common and ordinary". This highlights the difficulty in accurately generating surprise responses and suggests directions for future improvement.