Semantic and Topic fused Multimodal Transformer

Abstract

The human behaviors observed in multi modalities are naturally related with each other and many AI/ML methods try to catch such implicit relationships to enhance the prediction accuracies on users' behaviors. Ordinal methods, however, do not try to use the explicit relationships (e.g. taxonomy of emotions) across the modalities. In this paper, we suggest a new hypothesis for measuring the relevances across the modalities to improve accuracies.