Anomaly detection of road attachment facilities and vegetation using in-context learning with a multimodal large language model from vehicle-mounted camera imagery

Abstract

In response to the ongoing shortage of skilled engineers resulting from Japan’s declining birthrate and aging population, the implementation of AI-assisted inspection systems has become an urgent priority in expressway maintenance. Conventional AI-based approaches typically involve constructing dedicated models for detecting anomalies in specific targets, such as road attachment facilities or vegetation. However, given the wide variety of anomaly types, developing and maintaining separate models for each case presents significant practical limitations. In this study, we apply a multimodal large language model to anomaly detection from in-vehicle camera footage, aiming to identify multiple types of anomalies on expressways, including those involving roadside infrastructure and vegetation, using a single model, and verify its effectiveness. The effectiveness of the proposed method is further evaluated through experiments using real-world in-vehicle footage provided by East Nippon Expressway Company Limited.