2024 Volume 5 Issue 2 Pages 95-105
Application of AI(Artificial Intelligence) technology in infrastructure management is advancing. This article reviews recent advances and state of the art of the field from articles published in Japanese journal of “Artificial Intelligence and Data Science” issued by JSCE(Japan Society of Civil Engineers), with emphasis on use cases in image recognition, language models, prediction, acoustic data, monitoring, nondestructive evaluation, and combination with physics. Open data related to inspection record are pushing the progress of data driven approach in Japan. Inspection support by image data and interpretation of natural language are in progress by development LLM(Large Language Model). Structural monitoring is advancing by introduction of machine learning and CNN(Convolutional Neural Networks) technologies. Combination with physics is expected to improve clarity and quantification of AI.
Application of AI(Artificial Intelligence) in infrastructure management is advancing. This article reviews recent advances and state of the art of the filed from articles published in Japanese journal of “Artificial Intelligence and Data Science” issued by JSCE(Japan Society of Civil Engineers), focusing various use cases in infrastructure management.
Open data available in Japan related to infrastructure management is briefly introduced in chapter 2, and application of major AI technologies is reviewed in the subsequent chapters: chapter 3 on image recognition; chapter 4 on language models including LLM(Large Language Model); chapter 5 on prediction by machine learning; chapter 6 on acoustic data processing; chapter 7 on monitoring and nondestructive evaluation; and chapter 8 on combination of AI with physics including PINNs(Physics Informed Neural Networks).
Open data policy in Japan is pushing the advance of AI application in infrastructure management. Ministry of Land Infrastructure, Transport and Tourism of Japan is constructing open data platform for civil infrastructure1). Highway structures are inspected once in five years and the data of inspection record are openly available2).
Local governments are also moving toward to make inspection data as open data. Inspection data management system for periodic bridge inspection in Yamaguchi Prefecture is reported, and the importance of data cleaning is observed due to files with incorrect names and missing files3). In addition, a method of distributing inspection data as machine-readable open data on the Internet is proposed, and efficient generation method and distribution of open data are developed for local government4).
Image recognition is one of the most popular AI technologies, which is widely applied to detect or categorize damage in inspection. Image recognition can be categorized into three groups as shown in Fig.1: i.e., image classification; object detection; and semantic segmentation. In this chapter, image recognition application is reviewed in this order, and extension to diagnosis and multi-modal data is explained at the last part.
(1) Image classification
Image classification by convolutional neural networks (CNN) is applied to evaluation of corrosion and paint film deterioration of steel girder bridges5), which are mesh-divided into multiple images from the results of road bridge inspections. Improvement of performance by preprocessing, such as, flip horizontal, contrast enhancement, contrast reduction, and histogram flattening are tried6). Fig.2 shows examples of classification of surface state of steel bridges: i.e., corrosion, paint film deterioration, sound and others such as concrete surface.
Similar approach is also applied to concrete bridges7). Fig. 3 shows typical damage pattern of concrete bridges.
The inspection and diagnosis of weathering steel bridges is based on the deterioration of anti-corrosion function, which is evaluated with the corrosion condition rating by observation. In order to perform an accurate evaluation by the rating, inspectors need to have adequate experience. Fig. 4 shows grades of surface condition, where condition is better for increasing grades. It can be seen the classification is not easy for unexperienced engineers, and actually, the results vary by person. Many researches are found in identification of rust condition by AI using digital images of surface of weathering steel bridges8-12).
Surface preparation is required in painting or repair of corroded steel structures. The judgement of surface preparation grade for steel structure is performed through visual inspection. Therefore, there is a need to evaluate the surface preparation grade quantitatively. A support system that utilizes Vision Transformer to determine the surface preparation grade is reported13).
Because number of annotated rust images are relatively small, the possibility of rust image generation by GAN(Generative Adversarial Networks) is examined14). Performance of the support system to classify the surface preparation grade can be improved by enhancing the training images with images generated by GAN15).
(2) Object Detection
Various applications of object detection can also be found.
Owner-unknown bridges cause management difficulty, which leads to occasional accidents by lack of appropriate maintenance and repair. High-resolution aerial photographs and geospatial information are employed to detect bridges directly using deep learning16).
Damage detection using UAV photos is studied for various structures. Detection of corrosion and exposed steel bars in port structures17) and multiple damage patterns for bridge structures18) are studied. Since background of structures in photos, which is inevitable especially UAV based photos, is known to affect and reduce accuracy of AI considerably, special attention is paid to eliminate the effect of background18).
Damage detection of bolts and nuts in tunnel luminaire fixture components are reported19)．Fig. 5 shows an example image of detected damage.
Improvement of efficiency of maintenance of the headrace tunnel is desired because large machine is difficult to deploy in tunnel and the tunnel needs to be dried during inspection. Chalking points on the inner wall of the headrace tunnel are detected20).
(3) Semantic Segmentation
Typical application of semantic segmentation in infrastructure management is to obtain surface damage area or crack size in inspection.
Semantic segmentation is applied damage of concrete structures, and also degree of damage is evaluated21).
Application of semantic segmentation to corrosion of steel structure tends to be more difficult, because characteristics of color or boundary of corrosions are unclear in many situations. Improvement of accuracy of boundary is studied by transfer learning22). Damage on the inner surface of the sewer pipes is identified by semantic segmentation23).
Quantitative evaluation of crack length and width is investigated by semantic segmentation24)-26). Fig.6 shows an example of semantic segmentation of damage in a revetment24). Identified crack area is also used to assess degree of deterioration27).
For the semantic segmentation of damage, a large number of supervised images specifying damage at the pixel level are necessary, which requires time and effort. Methods for the semantic segmentation of cracks while reducing the cost of creating training dataset are developed 28)29). Quantitative evaluation of corrosion areas of streel bridges is important to decide action of countermeasures, such as repainting. Area estimation without pixel-by-pixel corrosion detection such as semantic segmentation is developed by combination of image classification by CNN and bivaluation of classified images to reduce this effort30).
CNN may be biased towards recognizing surface textures of objects. Effect of difference of texture is investigated by calculating features of crack images for revetment31). Since UAV images have a large proportion of background, which reduces accuracy, improvement by background reinforced training is studied32).
(4) Diagnosis and Multimodal Data
Evaluation of damage consists of determination of damage type and damage level.
Estimation of damage level for spalling, rebar exposure on concrete slab and crack on steel main girder by deep learning is investigated33). The method to support inspection personnel to evaluate damage level by AI is studied by constructing prototype and demonstration through field trials34).
Grad-CAM(Gradient-weighted Class Activation Mapping) is applied to verify the features of image, which contribute on the damage level determination35). Multi-task classification that classifies both the damage type and deterioration level at the same time is studied employing attention in evaluation36). Classification of the damage level without assigning a damage type in advance is proposed37) by training a single model through loss minimization considering the damage type and deterioration level.
Text data indicating the locations and materials of damage is introduced to image based deep learning and improvement of classification performance is reported38). Alkali Silica Reaction in concrete bridges from image data is determined by CNN incorporating additional information, such as aggregates origine, type of structural components, and so on39).
It is important to refer to past or similar damage images in the bridge diagnosis. Hence, similar image retrieval is studied by several authors. LSTM network was constructed to semantically extract damage features from captured images40). Graph neural networks is developed to learn the relationship between the individual images and to acquire feature representations of the record data. Similar record retrieval is tried using this result41). To annotate damage photos, similar image retrieval is applied across multiple datasets by introducing contrastive learning, regardless of the presence or absence of labels42).
Motion data of inspectors are also used to assist inspection, and extraction of tacit knowledge from experienced engineers is studied. Extraction of tacit knowledge from images by focusing on the line-of-sight information when diagnosing the soundness of bridges is attempted43).
A method for classifying inspection actions of engineers from inspection videos for the evaluation of damage is proposed 44). Damage detection method using egocentric videos with the aim of increasing the discovery rate is studied45),46). A method of classifying the skill level and visualization of its key factors to support the skill transfer is also presented47), which employs a graph convolutional network introducing a novel attention mechanism for the classification and visualization.
Use of language models is developing rapidly. Application of visual question answering (VQA) and LLM for inspection support is studied. Since, typical inspection records of damage consist of photos and related description, interpretation of photo images is especially attracting research interest.
Past inspection records are in paper form or the data is saved in PDF format, and text information in the bridge drawings of the inspection record is extracted utilizing object detection by deep learning48).
A deep learning model that can automatically generate sentences describing damage status based on images of various damages and structural members is developed incorporating attention mechanism49). Information from description of findings by inspectors, which are considered to describe the basis for judgment in damage diagnosis is extracted, and used as a dataset created to construct a deep learning model. The model is developed to answer questions related to the diagnosis of various components and damage in images50). Web system to support bridge inspection utilizing image captioning technology is developed51).
Incorporation of expertise to LLM through fine tuning using documents related to inspection is attempted52). Few-shot learning of the visual language model to generate description of findings is studied using damage images obtained by similar image retrieval of past inspection record53). Application to dialogue of technologies54), generation bridge inspection report55), and detection of discrepancies in inspection record56) are reported.
To evaluate the capability of language models in the field of civil engineering, dataset is constructed and evaluation index are proposed57).
Decision tree based methods are extensively used in prediction of condition or deterioration of structures.
Effect of environmental conditions and damage or initial defects on deterioration possibility is studied58). Application of GBDT(Gradient Boosting Decision Tree) for estimating the cause of damage and the repair method from the information of bridge management records is studied in order to assist the decision-making in selecting repair method59). Prediction of the deterioration of light fixtures installed in highway tunnels60), corrosion on the inner surface of the pipelines61), the adhered salt62) and the extent of crack damage63) in concrete bridges are reported.
A model to predict progress of deterioration is developed using GBDT. Inspection data of bridges managed by Tochigi Prefecture and GIS data such as climate and topography are combined using CNN to obtain training data64). Fig.7 shows the prediction shown on the map of the prefecture.
Acoustic diagnosis is often employed to assess damage and AI is also applied to acoustic data analysis.
Damages of steel finger type expansion joints of bridges are detected based on passing sound measured in a vehicle by comparison of past data of the same location before damage65). Detection and prediction of growth of cracks in lighting pillars are attempted from propagation characteristics of audible sound from excitation66).
Impact echo method is a popular technique to detect internal defect of concrete structures. However, the accuracy depends on the skill and experience of the engineer. Impact echo measured at site as Fig.8 is applied to autoencoder67). Deep learning using time-frequency characteristics is also studied68)69). Damage detection by classification of images obtained by spectrogram conversion of sound70), and Mahalanobis’ distance between the normal sound data and the test site obtained from feature vector extracted by CNN71) are reported. Local outlier factor method is applied to various experimental conditions72)-75). GAN is applied for data augumentation76)
AI is applied to evaluation of structural performance from data obtained from monitoring and nondestructive testing. Image recognition is also applied to monitoring problems.
(1) Monitoring
Application of image recognition for monitoring of structures is developed. Measurement method using video images for bridge reflection from long distance77) is studied and super-resolution technology78) and vibration compensation79) are applied to enhance accuracy. Feasibility of the frequency identification and anomaly detection of highway pole structures from video is investigated80). Possibility of satellite-based SAR(Synthetic Aperture Radar) image for monitoring of bridges is investigated using past images of collapsed bridge by identifying displacement time history81).
GNSS(Global Navigation Satellite System) is applied to bridge monitoring for a truss bridge82).
(2) Nondestructive Evaluation
The goal of NDE(nondestructive evaluation) is to evaluate the presence of defects inside structures, including their location, and size82).
In ultrasonic testing, AI is used to determine the location and characteristics of defects by analyzing the scattered waveform data obtained from measurement experiments and simulations83),84).
The infrared thermography method takes a thermal image of the concrete surface and detects internal defects86-88). Since infrared image can be taken remotely, use of UAV is considered86). Fig.9 shows an example of detected damage from infrared thermal image88). Self-supervised learning is applied88).
Electromagnetic radar image is also employed to detect internal damage of concrete structures89-91). Because radar image with internal damage is limited in number and in characteristics, GAN(Generative adversarial networks)90), and simulation91) are incorporated to enhance training data. GPR(Ground penetrating radar) images are used to detect underground inspection, and YOLO(You Only Look Once) is applied for real time evaluation92). Stagnant water in bridge deck slab, which accelerate deterioration, is detected by radar images93).
To make judgement on structural performance based on appropriate evidence, combination of AI with physical knowledge is developing.
Residual structural performance of corroded steel pipe pile pier is evaluated by AI to estimate both the area ratio and distribution image of damaged beams in comparison with structural analysis94).
Probabilistic framework for estimating residual strength of corroded RC structures using the observed corrosion crack width distribution and LSTM(Long Short-Term Memory) is proposed95). Two-dimensional steel corrosion distribution is estimated given the distribution of corrosion crack width using pix2pix. Considering model uncertainty involved in the prediction of steel corrosion, probabilistic density function associated with flexural capacity of aging RC beams is estimated96). Probabilistic load bearing capacity estimation of deteriorated RC members is extended including information on the width of corrosion cracks obtained by UAV photos97).
Surrogate models, which reproduce the inputs and outputs of physical phenomena in a data-driven manner, are increasingly being used as an alternative means of making fast evaluation of physical problems. Steel corrosion evaluation through the images is attempted in this manner based on data obtained from the corrosion tests, which provides relationship between the exterior appearance and the gained weight of the corroded steel specimens98). Surrogate model is developed to estimate the strength of PC-T girders damaged by salt attack with carbon sheet reinforcement99).
PINNs(Physics-Informed Neural Networks) are proposed, which solve the governing equations in a data-driven manner by introducing a loss function that represents the constraints imposed by the governing equations. Formulation and code implementation of PINNs for a one-dimensional continuum free vibration problem are available100).
Application of AI in infrastructure management in Japan is reviewed with emphasis on use cases in image recognition, language models, prediction, acoustic data, monitoring, nondestructive evaluation, and combination with physics.
Open data related to inspection record are pushing the progress of data driven approach in Japan.
Inspection support by image data and understanding through natural language are in progress by development of LLM. Data driven approach in prediction and monitoring is also advancing by introduction of machine learning and CNN technologies. Combination with physics is expected to improve clarity and quantification of AI. Challenges for practical application still remain.
For instances, further improvement of accuracy and robustness of image recognition of damage is required for implementation and automation in fields. Use cases of LLM and multimodal AI should be explored considering rapid development of the technology. Incorporation of expertise to pretrained models will expand application of AI to wider engineering problems. Reliability, accuracy and efficiency are critical AI based simulation and monitoring. To achieve these goals, establishment of evaluation criteria for accuracy and reproducibility, as well as datasets and corpora in civil engineering field are essential and need more attentions.
This work was partially supported by JSPS Grant-in-Aid for Scientific Research Grant Number 23H00198, 22H01561, and 21H01417
