Abstract
This paper presents an integrated system designed to translate Japanese industrial sign language into robotic commands, addressing significant accessibility challenges within manufacturing environments. We introduce a foundational dataset specifically tailored for Japanese industrial sign language and develop a real-time recognition system based on a bidirectional LSTM, architecture, enhanced by MediaPipe, landmark extraction, achieving an accuracy of 92% for single commands. The architecture of our AI agent, leverages large language models to convert recognized signs into semantically structured instructions, which are subsequently transformed into precise robot arm coordinates through our Vision-Language Model (VLM) calibration methodology. This system lays the groundwork for natural and accessible human-robot interaction via sign language, promoting workplace inclusion for hard-of-hearing individuals while ensuring operational efficiency in industrial settings. This paper emphasizes the sign language recognition methodology, highlighting its crucial role in facilitating accessible human-machine interfaces.
