2026 Volume 33 Issue 2 Pages 658-690
To improve the performance of Japanese large language models (LLMs), we organized a tuning competition that systematically examined a wide range of methods, including fine-tuning. The competition introduced two benchmarks: a mathematics task to evaluate reasoning ability and a safety task to evaluate the balance between safety and usefulness. Participating teams competed with models independently developed based on Japanese LLMs. In the mathematics task, using high school-level benchmarks, we measured the reasoning capabilities of Japanese LLMs and confirmed that leveraging outputs from powerful external models and supervised learning with large-scale synthetic data were effective. On the other hand, the lack of mathematical foundational knowledge in existing Japanese LLMs revealed that lightweight fine-tuning or reinforcement learning alone was insufficient for substantial improvement. In the safety task, we evaluated models using a Japanese safety-usefulness dataset, showing that methods such as Direct Preference Optimization and model merging contributed to improved safety. However, providing appropriate responses to seemingly harmful but actually answerable ambiguous cases remains a challenge. This competition, the first of its kind at a large scale for Japanese LLMs, yields insights from its design, implementation, and analysis that support further advances in Japanese LLM performance and inform the design of future competitions.
