Mathematical Analysis and Design Based on Expressive Power of Deep Neural Network with Residual Skip Connection

Abstract

Abstract. Model design is one of research topics in deep learning. Proposing a better model has been extensively studied, but there is no systematic theory of model design and model structure yet. In this research, we compare the structures of ResNet and DenseNet with a view to systematically understand the skip connection which is one of the structure of the model. As a result, it was theoretically confirmed that it is only due the regularity of full connected layers that gives differences in the expressive power of both models.