Evaluation of Treatment Progress Summaries Generated from Medical Treatment Records Using a Large Language Model

Abstract

It is urgent to improve the work efficiency of physicians because of the increase of their workload due to the declining birthrate and aging population, and a new overtime regulation that will be introduced in April 2024 as part of workstyle reform for physicians. We found that writing patient referral documents is a typical clerical work causing overtime work in the clinical observation, and developed a prototype system that supports physicians to create treatment progress summary using our Large Language Model. We evaluated the quality of draft summaries by the LLM and manually amended summaries with the help of ten physicians in the hospital. The averaged scores of draft summaries and amended summaries in ROUGE-1, ROUGE-2, and ROUGE-L were 46.6 and 42.9, 21.8 and 22.7, and 29.5 and 29.72, respectively. The draft summaries were natural and accurate according to physicians’ subjective evaluation. These results indicate that the proposed system has the potential to improve doctors’ work efficiency.