Emotional Space Using Supervised Crossmodal Contrastive Learning with Human Rating Distribution

Abstract

This study aims to acquire a mathematical representation of emotions from sensor data as a data-driven approach to emotion modeling. In order to represent human emotions, a modern representation learning method is used for embedding multimodal expressions into a shared latent representation (an emotional space). The proposed method uses supervised contrastive learning to embed emotionally similar data pairs closer together and dissimilar pairs farther apart, regardless of modality. Human emotions do not fall under any particular category but might be a complex mixture. Therefore, we consider a rating distribution with multiple raters' ratings for a single video, treat it as soft labels, and augment the loss function of supervised contrast learning using the similarity between soft labels. In the experiment using audio-visual data, we evaluate the robustness of emotion recognition when the modality is missing and confirm that the proposed method obtains shared representations of emotions across modalities in a low-dimensional emotional space. We also visualize the emotional space and observe the allocation of non-emotion-related information, such as the gender of the actor, to evaluate the effectiveness of the proposed method in representing the semantic relationships of human emotions.