Evaluation of Instruction Tuning on Finance-Specific Large Language Models

Abstract

It is beginning to be reported that small language models specialized for specific domains exceed the performance of general-purpose large language models. However, open-source language models specialized for the financial domain are limited, and language models need more evaluation with sufficient performance. Therefore, in this paper, we used benchmark sets containing various financial domain tasks such as sentiment analysis, classification, and question answering, and evaluated the performance change of a small chat model when subjected to multiple conditions of instructional tuning. We trained 7B and 13B models for this task by fine-tuning using low-rank adaptation. We empirically found that each model tended to improve the performance with both continuous pre-training and supervised fine-tuning despite over-fitting, and the generated results were affected by the instruction template.