Continual Pre-Training of Python Language Model to mT5

Abstract

Recently, Transformer-based pre-trained language models have achieved great success in natural language processing methods. As a result, there is a growing interest in software development and programming to apply pre-trained language models to a large amount of programming code. For example, CodeT5, which is T5 pre-trained with programming code, has shown significant performance improvements in various software development tasks such as code generation, code summarization, and code classification. However, building these models requires a huge amount of computing resources and time, and not everyone can do it. We propose a method of continuted pretraining to multilingual T5 for adapting python. This study reports that the proposed method shows improved performance results in tasks such as code generation and error diagnosis.