Brain Activity During Turn-Taking Using Virtual Characters: An fMRI Study on Humans Interacting with Virtual Characters

Abstract

In this study, we examined turn-taking behavior by measuring brain activity using functional magnetic resonance imaging (fMRI). We developed virtual characters that exhibited turn-taking behavior. These characters displayed facial expressions that indicated a willingness to continue talking, and regulated its timing in conversational turn-taking. We then confirmed the validity of these behaviors through behavioral experiments that used a pseudo-conversation between these virtual characters and human participants. In addition, we conducted an fMRI experiment where the participants were required to read part of a scenario in a conversation with the character. The results showed that the insular cortex and the right superior temporal gyrus were significantly activated when the character behaved properly in turn-taking. These results suggest that the activation of the right superior temporal gyrus was related to mutual understanding. The insular cortex has been shown to be related to empathy in previous studies. Thus, turn-taking may have a significant role related to mutual understanding and empathy in conversation. Our experimental design and findings may provide an objective framework for virtual character design, and contribute to the study of human conversation.