Systematic design of artificial deep neural networks based on scaling laws in signal propagation

Abstract

For more environmentally sustainable development of deep learning (DL) technologies, computational burden for tuning DL architectures should be reduced. This calls for more systematic strategies for finding an optimal set of hyperparameters which achieves a good balance between training speed and generalization performance. As a preliminary step toward this goal, we address the problem of how to tune fully-connected feedforward perceptrons in the so-called ``kernel regime'' in a systematic manner. By combining the existing theoretical tools, such as the Neural Tangent Kernel (NTK), and the analogy of the signal propagation dynamics with absorbing phase transitions, we conduct thorough analysis of the training dynamics of the neural network, including the case with finite depth. As a result, a simple strategy for optimally tuning the initialization hyperparameters and the depth is proposed.