An Impact of Weight Initialization on Model Evaluations in Neural Architecture Search

Abstract

Architecture is one key factor determining neural networks' performance, and neural architecture search, which aims at finding competent architectures without human effort, is one of the most intensive research areas of automated machine learning. While most papers in the area focused only on architecture, recent research show performance of architecture depends on other hyperparameters such as learning rate, and simultaneous optimization of them is needed to obtain a better model. This research focuses on weight initialization methods and investigates their impact on the performance of architectures after training. Through experiments on the architectures defined in NAS-Bench-201, we found an initialization method considering architecture significantly improved the performance of many models.