Overfitting characteristics of four-layer-deep-neural-network-based nonlinear equalizer for optical communication systems

Abstract

We compared the characteristics of overfitting in nonlinear equalizers based on a three-layer artificial neural network (ANN) and a four-layer deep neural network (DNN) for nonlinearity compensation in optical-fiber transmission systems. The characteristics were investigated using training data of a pseudo-random bit sequence (PRBS) and a finite-length repeated random sequence, with a varying number of input- and hidden-layer units in the ANN and DNN. The results showed that the DNN-based nonlinear equalizer had stronger overfitting characteristics for both the PRBS and random sequence, compared to the three-layer ANN-based nonlinear equalizer.