Improvement of Mini-Batch Size Dependency in Deep Learning for Reduction of Required Machine Resources

Abstract

In deep learning, batch normalization, which is commonly used to improve training performance, is recommendedto be used in conjunction with large mini-batch sizes during training on large datasets. However, increasing mini-batch size leads to an increase in required machine resources. Therefore, by reducing this mini-batch size dependencywhen adopting batch normalization and thereby reducing the required machine resources, we aim to alleviate thebarriers to exploring deep learning and promote diversification in its application scenarios. To this end, we proposea method that combines modified batch normalization with weight standardization to achieve training resultssimilar to those obtained with large mini-batch sizes, even when small mini-batch sizes are used. We demonstratethat our proposed method improves the problem of mini-batch size dependency compared to existing methods.