A study of multimodalization of spatio-temporal neural network for prediction

Abstract

These days, many novel neural networks for modeling spatio-temporal relationship are proposed as many kinds of spatio-temporal datasets like location dataset or traffic dataset are published and utilized. However, novel networks have a common problem that they cannot handle properly multimodal data with complex (multi-step) relationship, e.g. Modal A affects modal B and modal B affects modal C. This problem must be solved because much more kinds of spatio-temporal data will be distributed in the future. In this paper, for discussing what kind of structure “multimodal spatio-temporal network” should be, we conduct some preliminary experiments which includes extending existing spatio-temporal network to handle multimodal data and comparing prediction capability with the original network. Based on the results, we conclude that “multimodal spatio-temporal network” should properly encode the information which affects relationship of modals dynamically, e.g. meteorological data.