Construction of Human-in-the-Loop Inference for Automatic Structuring of Bridge Attribute Data Using Multimodal Large Language Models

Abstract

To advance infrastructure maintenance and management, extracting attribute information from non-standard bridge drawings remains a critical challenge. This study constructs a Human-in-the-loop (HITL) inference framework designed for the automatic structuring of bridge attribute data by leveraging Multimodal Large Language Models (LMMs). The proposed pipeline is built upon a Third Normal Form (3NF) database to eliminate update anomalies, integrating knowledge-retrieval techniques such as Retrieval-Augmented Generation (RAG) and few-shot prompting with master data correction. We analyzed the functional requirements for alignment with civil engineering databases and demonstrated that a "Two-Stage HITL workflow"—which separates the extraction of span IDs and structural attributes—significantly enhances inference accuracy.