Nonlinear Theory and Its Applications, IEICE
Online ISSN : 2185-4106
ISSN-L : 2185-4106
Special Issue on Recent Progress in Nonlinear Theory and Its Applications
Statistical neurodynamics of deep networks: geometry of signal spaces
Shun-ichi AmariRyo KarakidaMasafumi Oizumi
Author information
JOURNALS FREE ACCESS

2019 Volume 10 Issue 4 Pages 322-336

Details
Abstract

Deep neural networks are highly nonlinear hierarchical systems. Statistical neurodynamics studies macroscopic behaviors of randomly connected neural networks. We consider a deep feedforward network where input signals are processed layer by layer. The manifold of input signals is embedded in a higher dimensional manifold of the next layer as a curved submanifold, provided the number of neurons is larger than that of inputs. We show geometrical features of the embedded manifold, proving that the manifold enlarges or shrinks locally isotropically so that it is always embedded conformally. We study the curvature of the embedded manifold. The scalar curvature converges to a constant or diverges to infinity slowly. The distance between two signals also changes, converging eventually to a stable fixed value, provided both the number of neurons in a layer and the number of layers tend to infinity. This causes a problem: When we consider a curve in the input space, it is mapped as a continuous curve of fractal nature, but our theory contradictorily suggests that the curve eventually converges to a discrete set of equally spaced points. In reality, the numbers of neurons and layers are finite and thus, it is expected that the finite size effect causes the discrepancies between our theory and reality. Further studies are necessary to understand their implications on information processing.

Information related to the author
© 2019 The Institute of Electronics, Information and Communication Engineers
Previous article Next article
feedback
Top