Absorbing phase transitions in artificial deep neural networks

Abstract

For wider use of deep learning (DL) in society, deeper understanding of fundamental principles underlying various DL architectures is needed so that the users have better control over what they are actually doing with DL technologies. The deeper understanding is also likely to be useful for developing more environmentally friendly learning methodologies. As a preliminary step toward this goal, we study fundamental properties of signal propagations in artificial deep neural networks in this paper. More specifically, we show that there is a strong analogy between the signal propagation process in appropriately initialized fully-connected/convolutional deep neural networks and the dynamics of the so-called "absorbing phase transitions (APTs)" which can be found in some physical systems driven far away from equilibrium. We discuss, with numerical results on the signal propagation process, how these neural networks can be placed in a context of the theory of APTs and what theoretical/practical implication can be gained beyond the well-known mean-field theory.