Abstract
We assume that neurons have a strongly non-degenerate property in the mean including differentiation, express 3-layered neural networks as neurovarieties in function space with an inner product, and define a gradient flow of error-functions by the inner product. We study the gradient flow in the regular points area of the neurovariety. As a result, it is proved that Hessian matrices at the fixed points are non-degenerate and the fixed points are isolated, for almost every target function which is not on the neurovariety. Furthermore we can say that long (serious) plateaus arise only in the singular points area of the neurovariety.
