Structure from segmented motion for bridge 3D damage detection using UAV, AI, and MR
2023 年 4 巻 2 号 p. 27-34
詳細
2023 年 4 巻 2 号 p. 27-34
Periodical bridge inspection is essential to monitor the deterioration and maintenance progress. However, traditional inspection method requires a lot of works, expensive equipments, and time costly, and its difficult to implement periodically specially in developing countries. Therefore, this study proposed a Structure-from-Segmented-Motion (SfSM) method to enhance the bridge vision based inspection process. This method can localize and visualize the damage location in the bridge by utilizing various technologies such as UAV for data gathering, deep learning methods for damage segmentation, and Mixed Reality (MR) for digital transformation (DX). Firstly, optimal flight path for a single span I-girder bridge using UAV was proposed. This helps to fasten the visual bridge inspection, reduce workforce, and inspect some difficult to access parts without the use of expensive equipments. Then, the trained Deeplabv3+ was used to segment the corrosion damages of the images gathered. Finally, the 3D bridge model with and without the segmented corrosion were reconstructred using SfM and SfSM to visualize the location of damage in the whole bridge and viewed in a mixed reality (MR) platform. The proposed method will help the engineers to evaluate the bridge condition remotely which will save time and makes it safer.
The ageing of bridges takes toll on the economy of the country and the safety of the people. In Japan, periodic infrastructure maintenance and inspection is one of the largest challenges. Japanese government is encouraging engineers and researchers to innovate and use current technological advancement such as AI, IoT devices, etc. to enhance the maintenance process [1]. In line with this, researchers and industries have been proposing different methods to improve the traditional visual inspection method such as using different deep learning methods for damage detection, using UAV for data collection, creating digital twins, and exploring the mixed reality environment.
For bridge damage identification using various deep learning methods, researchers commonly conduct damage detection or damage segmentation. Dang et.al (2021), proposed to use YOLO to detect multi-type of bridge damages by enclosing the damages by bounding boxes [2]. Then, Sal and Dang (2021), proposed to use Mask-R-CNN to segment multiple bridge damages such as corrosion, cracks, spalling, etc. [3] Another method for damage segmentation was suggested by Liu and Dang (2022), which uses Deeplabv3+[4]. The trained deeplabv3+ model was adopted in this study. The usage of deep learning methods can reduce the labor cost and time of manually identifying the damages on each bridge inspection images.
In training the deep learning models, it is needed to create and label a large amount of bridge inspection images. Traditional bridge visual inspection requires a lot of manpower and heavy equipments to collect images. Narazaki et. al (2017)(2021), proposed a method in which a set of railway bridges component and damages synthetic data was created for deep learning training [5][6]. Dang et. al (2020) proposed to use the UAV inspection images for deep CNN bridge deterioration training[7]. Chun et al (2020) also proposed a framework of combining the UAV, artificial intelligence, and IoT devices to enhanced the bridge inspection[8].
Some of the obstacle in using UAV for bridge inspection are weather conditions, lighting conditions, GNSS denied areas in which manually controlled drones are needed, and other parts of the bridge that has narrow enclosed areas which are difficult to access by the drone. Montes et al. (2021) proposed a semi-autopilot UAV flight plath to inspect a GNSS denied area under the bridge deck using a low cost drone[9]. Then, Kim et al (2023) combined the AI bridge component recognition and images captured by the uav to decide the flight path automatically in a virtual environment[10].
Aside from damage detection and utilizing UAV for image collection, the reconstruction of 3D models was also considered by researchers. Kim et al (2020), used the bridge point cloud data using laser scanner to train a CNN model which aims to categorize the bridge components such as pier, background, and deck[11]. Inadomi et al (2021) used Structure-from-Motion (SfM) approach to create a bridge 3D model and used the point cloud data for bridge component segmentation[12]. Yamane et al (2022) used SfM to reconstruct the 3D model of the bridge then project the damages into the 3D model[13].
After the 3D model reconstruction, Dungston et. al (2005), elaborated the potential use of Mixed-Reality visualization to various application in civil engineering and construction[14]. Cuong et al (2020), proposed a method to project the BIM model and 3D Model reconstructed using SfM into Mixed Reality (MR) platform[15].
Therefore, this study proposed to combine different methodology such as UAV, deeplearning for damage segmentation, creating a 3D Model of the bridge using SfM and SfSM, and mixed reality platform. The structure-from-Segmented-Motion (SfSM) was proposed to visualize the distribution of damage in the bridge. The proposed method’s feasibility was tested in a 1-span I-girder bridge and the preliminary progress will be discussed in the following chapters.
To enhance the traditional bridge inspection, this study proposed a method on different stages of visual inspection and evaluation as shown in Figure 1. Firstly, the usage of Unmanned Aerial Vehicles (UAV) was proposed to gather inspection images from the bridge. Using drones will help to reduce the usage of heavy equipments to access some parts of the bridge. Then, after the inspection images were collected, the damages will be segmented using some trained deep learning model. Furthermore, the 3-D model of the bridge will be reconstructured using the raw images using Structure-from-Motion (SfM) and the images with segmented damages using the Structure-from-Segmented Motion (SfSM). In this case, the pixels of the segmented damages will be included in the 3-D model in which the damage location in the whole bridge can be observed easily. Lastly, the 3-D model will be saved digitaly and can be viewed using the mixed reality platform. Using MR platform, engineers can inspect and evaluate the bridge current status even on their offices. The past and periodically inspection results and 3D-bridge model can also be saved digitally to promote digitalization. The next chapters will discuss the data gathering procedures, damage segmentation, 3D model reconstruction using SfSM, and mixed reality platform.
This study uses UAV for data collection to inspect some difficult to access parts of the bridge without the use of other equipments and to get a suitable data for 3D model reconstruction. However, some of the difficulties encountered in the data gathering includes the following:
1. Environmental obstacles such as trees, lighting poles, electrical cables, birds, pipes along the bridge deck, etc. restrict the UAV’s flight path.
2. Strong wind and possible rain should be checked in advance.
3. The low lighting condition under the bridge deck will affect the quality of the reconstructed 3D model.
4. The UAV’s limited battery capacity should be taken in account in the flight planning.
5. Some bridge components such as cables or truss girder are difficult to inspect due to thin cross-section area or complexity.
The horizontal and vertical distances, and the shooting position should be planned carefully so that the images will be aligned with high accuracy during the 3D model reconstruction. Furthermore, this study suggest to have 70% image overlap to generate a more accuracte 3-D bridge model.
The low cost UAV used is DJI Mavic mini as shown in Figure 2. This drone can fly upto 30 minutes with a maximum speed of 29 meter / hour depending on the environments condition. It has an obstacle avoidance features on the downward direction and the maximum range is 4000 meters.
This study focuses on the top and side view of the selected bridge. The flight paths for the 1-span Igirder bridge are suggested in this research. The path planning is significant to optimize the usage of drone’s battery and reduce the time of inspection. For the top view of the bridge, the horizontal distance flight path was shown in Figure 3. The horizontal distance highly depends on the width of the deck and is not fixed to 1.5 meters. Three different angle variations were also considered as shown in Figure 4. The angle variations are important to capture the overlap of each images and to get more pixel informations. Then, the variations on the vertical distances are shown in Figure 5.
For the side view of the bridge, the vertical and horizontal distances are shown in Figure 6. The vertical distance varies depending on the railing height and bridge girders height. The horizontal distances also varies in case that there are some obstacles along the length of the bridge such as trees, post, etc.
The angle variations on capturing the side view of the bridges is shown in Figure 7. There angle depends on the vertical and horizontal distance of the UAV to the side surface of the bridge. These angles are recommendations on the selected bridge and may varies on other bridges with different dimensions and bridge types.
A one-span I-girder bridge was used in this study as shown in Figure 8. It was selected because of large areas of corrosion on the bridge railings which can illustrate the proposed method clearly.
A total of 950 images were collected and the sample images were shown in Figure 9. There are lot of corrosion damage areas in the bridge railing as observed. These images will be used to reconstruct the 3D bridge models and the corrosion damages will be segmented.
Using the collected images, the trained deeplabv3+ model [4] was used to segment the corrosion damages. The deeplabv3+[16] architecture consists of an encoder-decoder architecture as shown in Figure 10. The encoder learns the features of the target class and create a feature map. Then, the decoder regenerates the input image based on the feature map processed by the encod-er. The environment used was Pytorch version 1.9.0, the backbone used was Resnet-101cuda 10.2, GPU of Tesla P1000, and the platform was in googlecolaboratory pro using python as programming language.
The training dataset was collected from an actualbridge inspection report and previous UAV inspection as shown in Figure 11 and Table 1. It was annotated using an open source annotation software program called "VGG annotator". The total dataset used was 2290 images, which was divided to 1823 training images, 325 validation images, and 142 testing images. The batch size is 6, crop size is 78, learning rate of 0.001 with poly decay, cross entropy for loss function, and some fine tunings.
The training results was shown in Table 2, the parameters were calculated after 50 epochs. During validation, the pixel accuracy is 94.77%, the Mean pixel accuracy is 87.31%, the mean Intersection of Union (mIoU) is 80.81%, and the frequency weighted Intersection of Union (fwIoU) is 90.37%. The visualization of the training results are shown in Figure 12.
The visualization of the raw image and the image with segmented corrosion damages were shown in Figure 13. It can be seen that the corrosion damages were marked with green color. Then, to visualize the segmented corrosion more, Figure 14 shows other samples.
Structure-from-Motion (SfM) is one method to reconstruct a 3D model of an object using a collection of images. This study enhanced this method by introducing the Structure-from-Segmented-Motion (SfSM) method to reconstruct the 3D model of the bridge with the segmented damage in the images. Two 3D models were constructed which includes the 3D model using the raw images (SfM) and another with the consideration of the images with segmented corrosions (SfSM).
Using SfM method, firstly the feature points of the pixels will be aligned, then the point cloud and dense cloud will be generated. Lastly, the texture will be added and rendered the 3D model to make it more realistic. This study suggested to set a uniform color for the segmented damages so that each of the segmented damage pixels feature points will be aligned. The image overlap of 70% is highly significant on this process. For the first 3D model, the feature points alignment, point cloud generation, and bridge 3D model were shown in Figure 15, Figure 16, and Figure 17. Then, the final 3D model with rendered texture was shown in Figure 18.
Then, the proposed SfSM method was used to reconstruct the 3D model of the bridge with segmented corrosion. The feature points alignment, point cloud generation, and bridge 3D model were shown in Figure 19, Figure 20, and Figure 21. Then, the comparison of the reconstructed 3D model with and without segmented corrosion damages were shown in Figure 22. The location of corrosion damages in the whole bridge can be visualized easily. However, this approach is highly dependent on the accuracy of the trained damage segmentation deep learning model.
The proposed mixed reality platform uses Hololens2 as shown in Figure 23 as a tool. The person wearing this device can view the 3D model / Bridge Images and can rotate, resize, and move the object in the mixed reality environment. The sample MR interface was shown in Figure 24. The reconstructed 3D models were projected into MR environment and can be viewed by engineers even in their offices. It can also help to sort out large amount of bridge inspection paper reports and make the damage locations easily to visualize.
The proposed method utilized the use of UAV, Deep learning method for damage segmentation, Structure-from-Segmented-Motion (SfSM) to create bridge 3D model with projected corrosion damages, and Mixed Reality (MR) platform. The proposed SfSM method for creating a 3D model with visualization of the segmented corrosion damage on the bridge was proved to be feasible as shown on the preliminary progress of inspecting a 1-span I-girder bridge. The UAV flight path was suggested to have different horizontal and vertical distances and angle variations of the UAV to the bridge surface. The image overlap was proposed to be 70% to create a more accurate 3D model. The deep learning method for damage segmentation will be further improve. However, other types of damages such as cracks are difficult to be applied in the proposed SfSM method because the damage pixels will be few and the feature alignments will be difficult. Further research improvements includes the flight path planning for different type of bridge, creating an application inside the MR environment to save more inspection data, and usage of more advanced UAV equipment.
