Comparative verification of discrepancy detection capability of AI chatbot on xROAD road bridges data

Abstract

In the process of digitizing data in the civil-engineering field, transcription errors can occur when converting existing handwritten records into electronic form, resulting in mismatches between data formats. As digitization progresses, such inconsistencies are expected to continue accumulating; however, it is impractical to enumerate all possible conditions under which discrepancies may arise and address them algorithmically. While prior studies have examined various AI chatbot individually, no comparative evaluation using unified prompts and benchmarks on the latest models had been conducted. This study investigates how each AI chatbot’s inherent characteristics affect its ability to detect inconsistent data in xROAD, performing a comparative evaluation—particularly of ChatGPT’s o3 and o4-mini-high models. In addition to Claude (Opus4), which was considered in earlier work, Gemini 2.5 Pro under the same conditions are evaluated. At this stage, system-level issues make them difficult to obtain results comparable to those achieved with ChatGPT.