Bridge damage diagnosis using inspector’s findings with OpenAI CLIP

Abstract

Damage diagnosis for road structures in Japan is carried out once every five years, and is based not only on an objective evaluation of the damage conditions, but also on a comprehensive evaluation that incorporate the expert opinions of inspecting engineers. On the other hand, much of the research into using AI to improve the efficiency of inspection and maintenance has focused on objective facts, and it is possible to perform comprehensive damage diagnosis by using the expert opinions of inspecting engineers. Therefore, for language-based information, it is necessary to understand the complex relationships between rich expressions through the modeling of the relationship between images and text. In this paper, the bridge damage diagnosis with CLIP was investigated, using the findings and damage images recorded in xROAD and aiming at the inherit language-based information. Specifically, features were extracted from each of the findings and damage photographs listed in xROAD, and a VQA model for classification data was built by learning a feature vector that integrates these features. The results suggest that by using language-based information, such as the inspector’s findings, CLIP-based bridge damage diagnosis can improve generalization performance for deep learning-based tasks.