Explanation of Traffic Risks with LLM Using GIS Data and Street Images

Abstract

Consideration of traffic risk in driver assistance systems and automated driving technology is important in preventing traffic accidents. Traffic risks are considered to be contained in image information. However, it is difficult to explain traffic risk in driving scenes from image information alone, and research in this area has not yet progressed sufficiently. In this study, we propose a multimodal framework that can explain traffic risks by using GIS data and street images. This framework identifies the coordinates of high-risk areas from traffic accident risk maps created based on GIS data and trains a multimodal network using street images associated with those areas. By doing so, we construct a framework that effectively explains traffic risk in an arbitrary scene. Experimental results show that the proposed framework can generate captions that explain traffic risks for high-risk areas based on GIS data.