FinMegatron: Large Financial Domain Language Models

Abstract

General domain pretrained large-scale language models, such as BERT and GPT3, have achieved state-of-the-art results among numerous NLP classification and generation applications. This pretraining technology is also willing to be used in vertical domains, such as finance. The downstream applications include financial event extraction from news, summarization, and causal inferencing. In this paper, we propose large-scale pretrained BERT models for financial domain in English and Japanese languages. The original datasets come from professional financial news. We empirically study the factors of sub-word vocabulary set, model size and their impacts to the downstream financial NLP applications. The code and pretrained models are released from https://github.com/NVIDIA/Megatron-LM.