On the study of the memory-less quasi-Newton method with momentum term for neural network training

Shahrzad Mahboubi; Ryo Yamatomi; Indrapriyadarsini S; Hiroshi Ninomiya; Hideki Asai

doi:10.1587/nolta.13.271

Special Section on Nonlinear Science Workshop on the Journal

On the study of the memory-less quasi-Newton method with momentum term for neural network training

Shahrzad Mahboubi, Ryo Yamatomi, Indrapriyadarsini S, Hiroshi Ninomiya, Hideki Asai

Author information

Keywords: neural networks, training algorithm, memory-less quasi-Newton method, Nesterov's accelerated gradient, momentum term

JOURNAL FREE ACCESS

2022 Volume 13 Issue 2 Pages 271-276

DOI https://doi.org/10.1587/nolta.13.271

Details

Abstract

Quasi-Newton (QN) methods have shown to be effective in training neural networks. However, the computation and the storage of the approximated Hessian in large-scale applications is still a problem. The Memory-less QN (MLQN) was introduced as a method that did not require the storage of the matrix. This paper describes the effectiveness of the momentum term for the accelerated MLQN method through computer simulations on function approximation and classification problems.

Corresponding author

Register with J-STAGE for free!