Predicting Numerals in Text Using Nearest Neighbor Language Models

Abstract

The k-nearest neighbor language model (kNN-LM) is a language model that extends pre-trained neural language models with the idea of kNN search. In this study, we apply the kNN-LM to the masked numeral prediction task (MNP), which requires predicting masked numerals from their contexts, to improve the accuracy. In our experiments, we confirmed that the kNN-LM achieves better accuracy than the base language model in the MNP. We also compared different context spans used for calculating context representations for the kNN search to improve the search accuracy by using only the words that are closely related to the masked numeral: only the mask and its surrounding words, and only the mask and its following words. As a result, we confirmed that the kNN search using only the embeddings of mask tokens for numerals is the most effective for the MNP.