An Analysis of Multimodal User Behaviors for Objective Evaluation of Spoken Dialogue Systems

Abstract

An objective evaluation method for spoken dialogue systems is argued. In conventional research, subjective evaluations are often used in user experiments. However, from the perspective of research comparison and reproducibility, objective evaluations are also necessary. Therefore, we propose a framework for indirectly but objectively evaluating systems based on multimodal user behaviors. In this report, we clarify user behaviors related to subjective evaluations in different dialogue tasks, including attentive listening, interviewing, and first-time conversation.