Visualization for Multimodal Relational Data using t-Stochastic Graph and Neighbor Graph Joint Embedding

Abstract

Multimodal relational data analysis has become of increasing importance in recent years, for exploring across different domains of data, such as text and images obtained from Flickr or Instagram. A variety of methods have been developed for visualization; to give an example, t-Stochastic Neighbor Embedding (t-SNE) computes low-dimensional feature vectors so that their similarities keep those of the observed data vectors. However, t-SNE is designed only for a single domain of data but not multimodal relational data; this paper aims at visualizing multimodal relational data, whose associations across domains and data vectors in some domains are observed. Our proposed method (1) first computes augmented associations for the multimodal relational data, where the associations across domains are observed and those within domain are computed via the observed data vectors, and (2) jointly embeds the multimodal relational data to a common low-dimensional subspace. Through Flickr dataset visualization, we demonstrate the promising performance of the proposed method.