Structure Construction and Performance Analysis of RNN Aiming for Reduction of Calculation Costs

Abstract

This paper presents novel RNN structures. In order to reduce the calculation as well as to overcome the analysis difficulty of gate structure and the problem of data dependence related normalization in the conventional method, 4 types of new RNNs inspired by shortcut connection were constructed while taking the advantages of gated RNNs and a sech function. Experiments and analyses are carried out to evaluate the performance of the RNNs. In detail, the experiments are performed using two corpus of WikiText-2 and IMDB, which have different properties. As a result, in the binary classification task using IMDB, the accuracy was about the same as LSTM and GRU with the parameter of 1/6 or less. However, in the language modeling task using WikiText-2, even if multilayer and the parameters of the intermediate layer were increased, the results were worse than that of conventional RNNs. This should be clarified in our future research.