Multimodal Semantic Prediction Utilizing Semantics and Latent Uttarance Topics based on Variational Auto Encoder

Abstract

In the field of multimodal machine learning, we are faced on the problem of how to combine multiple sources of input data to produce more accurate results than simply summarize the training results for each input data, anytime. Against this issue, we have developed a new model for multimodal sentiment analysis that superior to existing models for accuracy by using the following three elements: (1) applying semantics to each word, (2) extracting relationships between modalities using attention, and (3) adding topic information based on the latent space for the entire utterance that unifies the modality information.