Effects of prompt constraints on the procedural consistency and executability of LLM-generated DEM input files

Abstract

Scientific computing software is highly versatile, but preparing reliable input scripts remains difficult for novice users because syntax, modeling procedures, and solver-specific requirements must all be handled correctly. Large language models (LLMs) are increasingly used as effective tools for assisting script generation. However, because LLMs may produce different syntactic expressions and procedural choices in each generation, evaluating both consistency and executability is essential. This study investigated a gravity-driven particle-packing task using PFC2D, a discrete element method software package. Three prompts with different levels of constraint were each used to generate ten scripts. Increasing the level of constraint reduced variation in particle-generation methods and other modeling choices and increased the static fulfillment rate of specified requirements. Nevertheless, execution tests showed that only three, four, and one scripts completed successfully under P0, P1, and P2, respectively. Several failures were caused by initialization order, missing solver settings, particle-generation syntax, and software-specific FISH APIs. These findings demonstrate that stronger prompt constraints can improve procedural consistency and static specification compliance, but do not guarantee executable scripts. Therefore, LLM-generated CAE input scripts should be evaluated separately in terms of consistency, executability, and the validity of numerical outputs.