Mathematical Representation of Emotion Using Multimodal Deep Neural Networks

Effects of the Number of Dimensions of Emotional Space on the Performance of Recognition and Unification Tasks

Abstract

To emulate human emotions in robots, the mathematical representation of emotion is important for each component of affective computing, such as emotion recognition, generation, and expression. In a method that represents emotions by vectors of continuous values (Emotional Space), it is necessary to consider the number of dimensions of Emotional Space. In this study, we propose a method of integrating multimodalities on a DNN acquiring Emotional Space. We aim at the acquisition of modality independent Emotional Space by combining the emotion recognition task and unification task of Emotional Space of each modality. Through the experiments with audio-visual data, we confirmed in various dimensions of Emotional Space that there are differences in Emotional Space acquired from uni-modality, and the proposed method can acquire a modality independent Emotional Space. We also investigated the compatibility of the recognition and the unification score by changing the number of dimensions of Emotional Space.