The size of a hierarchical neural network is desirable to be compact according to each application problem, because the network size is closely related to not only the computer memory but also the generalization ability. If the network is too large, it will be over-fitting the training data and bad generalization will occur. In an opposite situation, training for its own sake will fail due to under-fitting.
This paper deals with a method to reduce the network size and to obtain a compact network equivalent to the original one. Principal Component Analysis (PCA) performs a central role in a size reduction. The reduction can be implemented in the follwing way: (a) At first, train a largish network in order to avoid under-fitting. (b) after training, the output matrix of hidden units is changed into the component score matrix by PCA. (c) determine n components corresponding to the n largest eigenvalues according to PCA so that the cumulative proportion may be almost 100%. Then the number of hidden units can be reduced up to n. (d) determine weights and thresholds by solving simple linear equations so that the reduced network may be equivalent to the original one.
The proposed method is applied to a simple pattern recognition problem and the result shows the effectiveness of the proposed method.
抄録全体を表示