Dependency-Based Self-Attention for Transformer Neural Machine Translation

Abstract

This paper proposes a new Transformer neural machine translation (NMT) model that incorporates dependency relations into self-attention on both the source and target sides, “dependency-based self-attention”. The dependency-based self-attention is trained to attend to the modifiee for each token under constraints based on dependency relations. This was inspired by linguistically-informed self-attention (LISA). LISA was originally designed for the Transformer encoder for semantic role labeling. However, this paper extends LISA to the Transformer NMT by masking future information on words in the decoder-side dependency-based self-attention. Further, our dependency-based self-attention operates with sub-word units created by byte pair encoding. In the experiments, our model achieved an increase of 1.04 and 0.30 points in the BLEU over the baseline model, respectively, on the Asian Scientific Paper Excerpt Corpus Japanese-to-English and English-to-Japanese translation tasks.