製品ドキュメントを活用した大規模言語モデルによるコード生成の品質向上に関する研究

抄録

This study proposes a method for automatically generating End-to-End (E2E) test code from product documentation using a large language model (LLM). The product documentation refers to materials such as manuals, tutorials, FAQs, and step-by-step operation guides that help users achieve their objectives with the application. The proposed approach aims to generate E2E test code with improved coverage and quality by leveraging specific and detailed instructions in the product documentation. We generate E2E test cases and subsequent test codes through separate prompts to the LLM, taking these documents as input. We used a web application with six functionalities in an experiment and compared different document types when evaluating test code generation. Results showed that product documentation-based tests achieved higher functional coverage, suggesting the advantage of detailed user operation guidance for deriving comprehensive test scenarios. By contrast, requirements documents, and user stories were effective for different functionalities subsets, pointing to each document type's complementary strengths. These findings indicate combining or enhancing documentation sources can produce more robust test coverage and improve overall software quality. This approach offers new insights into efficient software development practices that integrate LLMs with existing documentation resources.