Acquiring Bidirectionality via Large and Small Language Model

Abstract

In this study, we raise the issue of uni-directionality when applying large causal language models to classical NLP tasks. As a solution, we propose a method of utilizing the concatenated representations of a newly trained small-scale backward language model as input for downstream tasks. Through experiments in named entity recognition tasks, we demonstrate that introducing backward model improves the benchmark performance more than 10 points. Furthermore, we report that the proposed method is especially effective for rare domains and in few-shot learning settings.