Development of a 3D-visible Multimodal Attentive Listening System that Utilizes Posture Mirroring Effect

Abstract

Attentive listening is an effective communication technique for rapport building. Our project developed a 3D-visible multimodal attentive listening system. The proposed system projects a 3D avatar on a naked-eye stereoscopic display in order to effectively communicate non-verbal information such as posture mirroring effects. The proposed system has spoken communication based on listening dialogues, such as backchannels, empathy, repeats, modality-based responses, and mutual questions. Furthermore, the proposed system has multiple means of communication consisting of nonverbal information such as facial expressions, eye gaze, blinking, nodding, and posture mirroring. In this article, our project experimented with 60 participants, consisting of university undergraduate and graduate students. The participants freely talked for three minutes about hobbies, favorite/disliked foods, future goals, worries, how they spend their holidays, childhood memories, club activities, part-time jobs, failures, and other topics in a one-on-one room. The participants then responded to a 22-item sensitivity evaluation. Our results show that the avatar's listening attitude, such as the way she listens and shows empathy, was highly evaluated by the participants. Moreover, the majority of participants felt that their avatars were friendly and that they wanted to talk to them.