Spoken Dialogue Method Considering Nonlinguistic Emotion Expression of User

Abstract

In general, spoken dialogue systems respond to user utterances only according to text information of speech. However, speech recognition system usually recognizes only “what the user said,” but ignores “how the user said” like user’s emotion or intention, because it is often expressed by paralinguistic or nonlinguistic information. Even if the systems can estimate the user’s emotion, it is difficult to make appropriate reply according to both text information and emotion because vast number of rules should be prepared for combination of various utterances and emotions. On the other hand, statistical response method is robust for unexpected utterances, although, it requires learning data with emotion tag which is manually annotated. This paper proposes a spoken dialogue method according to both text information and emotion in a speech. The user’s emotion estimated from acoustic features is transformed into emoji, and the emoji is input to the statistical response system along with speech recognition result. The response system learns adjacency pairs that contain emojis which are collected from tweet-reply pairs on Twitter. The dialogue system replies to the user by a synthesized voice at the end of the process. When the output of response system contains emoji, speech synthesis system changes the tone of the voice considering the emoji. Experimental results revealed that the proposed method could make better responses considering the user’s emotion than the other methods which do not use nonlinguistic emotion expression of the user. Furthermore, speakers impressed more positive feelings to the proposed system than non-emotion methods (p<0.05).