Construction of pooling layer by skip connection and analysis based on expressive power of these models

Abstract

In this research, we consider structures such as pooling layer and skip connection from the viewpoint of expressive power in order to organize design of neural networks models. We showed that widely used these structures can be understood as a composition of affine functions and concatenated activation functions. Moreover, we show the followings: (i) the pooling layer explicitly decreases expressive power, (ii) there is no deference in expressive power between addition and concatenation as skip connection for fully connected neural networks, and (iii) the single activation block has superior expressive power compared to the multiple activation block. These results propose one guideline for design of neural networks models.