Stochastic Regularization for Residual Networks: Shake-ResDrop and Shake-SENet

Abstract

Recently, residual networks(ResNets) and their improvements, such as stochastic regularization, have proven to be able to reduce overfitting during training processes. In this paper, we propose two stochastic models which combine stochastic regularization and attention mechanism. The two models are based on ShakeDrop, combining either SENet or Stochastic Depth with ShakeDrop itself. Both of our methods were able to improve existing ShakeDrop results on CIFAR-100.