Length-constrained Neural Machine Translation using Length Prediction and Perturbation into Length-aware Positional Encoding

Abstract

Neural machine translation often suffers from an under-translation problem owing to its limited modeling of the output sequence lengths. In this study, we propose a novel approach to training a Transformer model using length constraints based on length-aware positional encoding (PE). Because length constraints with exact target sentence lengths degrade the translation performance, we add a random perturbation with a uniform distribution within a certain range to the length constraints in the PE during the training. In the inference step, we predicted the output lengths from the input sequences using a length prediction model based on a large-scale pre-trained language model. In Japanese-to-English and English-to-Japanese translation, experimental results show that the proposed perturbation injection improves the robustness of the length prediction errors, particularly within a certain range.