Symbol Emergence of Emotion through Multimodal Deep Generative Models and Metropolis-Hastings Naming Game

Abstract

According to the theory of constructed emotion, our brains actively form emotional categories by multimodal stimuli from inside and outside our own body and feel emotions through the prediction process of emotional categories from the stimuli.Research into the forming and cognitive processes of emotions is progressing from the perspective of the predictive coding of embodied information; however, there has been little research on the emergence of emotional symbols, i.e., how emotions are interacted with and shared between individuals. In this study, we examine the emergence of emotional symbols by modeling the interaction between two individuals that form categories from multimodal information based on one's own body using the Metropolis-Hastings Naming Game (MHNG). In our experiment using two agents that perceive visual, auditory, and interoceptive sensations, we verified the differences in the structure of emotional categories formed depending on the presence or absence of MHNG interaction. As a result, it was confirmed that each agent forms emotional categories from its own sensations, and the categories were affected by shared symbols among agents using MHNG.