Evaluating the applicability of an LLM-based agent in civil engineering

Abstract

Large Language Models (LLMs) are known to have poor response performance in highly specific domains such as civil engineering. Methods such as Retrieval Augmented Generation (RAG) have been proposed to address this problem and are increasingly being used in civil engineering. On the other hand, there may be problems that cannot be solved by current RAG, such as when the number of target documents is very large. Recently, there have been an increasing number of examples of LLM-based agents in various fields, their use is also expected in the field of civil engineering, e.g. for exhaustive knowledge search. In this paper, a comparative analysis of a RAG and an LLM-based agents is performed to evaluate their application. The results on this paper show that although the RAG method showed better performance, the LLM-based agent method was able to refer to more appropriate parts of documents than the RAG method, indicating its potential use in civil engineering. Finally, we summarised the issues that need to be addressed when using LLM-based agents.