Adaptive Structural Learning Method of Recurrent Deep Belief Network for Time Series Analysis

Abstract

Deep Belief Network (DBN), which is well known to be a kind of Deep Learning methods, has a deep network architecture that can represent multiple features of input patterns hierarchically. Each layer employs a pre-trained Restricted Boltzmann Machine (RBM). In DBN model including RBMs, we may meet the difficulties in finding the optimal network structure and the best set of weights and threshold values. For the solution, we developed the adaptive structure learning method of DBN that can discover an optimal number of hidden neurons for given input data in a RBM by neuron generation / annihilation algorithm, and hidden layers in DBN by the extension of the algorithm. Moreover, the Long Short-Term Memory (LSTM) model can make an accurate prediction for a time series data set. The network architecture of LSTM has been proposed in various ways and can represent high accurate prediction for the benchmark data set, but the problems related to the optimal structure and some parameters still remains. In this paper, the adaptive structure model of RBM and DBN is applied to the LSTM model and the effectiveness was verified by 10-fold cross validation on benchmark data sets.