重みの振動を抑制する階層型ニューラルネットワークの学習法

Kick Out法

抄録

Backpropagation algorithm (BP) having only a gradient term converges very slowly, because of the oscillation of weights occurring in regions where the error surface forms a ravine. In order to reduce the oscillation of weights, the momentum term was introduced. However, it has not worked well to reduce the oscillation because the gradient includes the component across a ravine which causes further oscillations. To overcome this problem, we should focus near the bottom of a ravine where the steepest descent direction is the same as the downward direction along a ravine. We described a method to correct the position of weights near the bottom of the ravine and proposed a new accelerated learning algorithm based on Jacobs' algorithm with it. The proposed algorithm reduces the oscillation of weights quickly and converges about 18 times faster than the standard BP in the problem of a sine function approximation.