On a partial differential equation based neural network

Abstract

A continuous formulation of a neural network based on a partial differential equation (PDE) is proposed. By assuming binary classification with cross-entropy, we formulate the neural network as an optimal control problem of a PDE. The existence of the optimal solution is theoretically proved. By finding the solution and optimal control in Hilbert spaces, we provide the explicit representation of the Fréchet derivative of the cost function at the solution. We also propose a gradient-descent-based algorithm that delivers a (sub-) optimal control.