Proposition of Multimodal Time Series Data Analysis Framework by CNN based on Multi-Channel Image Conversion

Abstract

In recent years, with the development of IoT and sensor technology, various data can be acquired. In this case, it is expected to establish analytical methods capable of extracting the characteristics of relevances of each variable of multimodal data. In this study, time series variables with different dimensions on the same time axis are converted to color change images as RGB which is the three primary colors of light, and Convolution Neural Network(CNN) is applied to this. Next, we propose a method to perform more effective feature extraction by converting the image using XYZ, Lab color space reflecting the color visual stimulus with RGB as the base. We compared accuracy with existing classification method and showed the effectiveness of the proposed method. Moreover, by converting time series in various color spaces. It is suggested that higher performance feature extraction can be realized than when processing each variable as independent.