マルチモーダル環境音認識に基づく擬音語変換と音声認識を用いた聴覚障がい者支援システムの試作

抄録

This paper focuses on onomatopoeia conversion based on multimodal environmental sound recognition, and reports on a proto-type system that uses this method to support the hearing-impaired. We proposed and prototyped a system using multimodal en-vironmental sound recognition-based onomatopoeia conversion and speech recognition. In this prototype system, text converted by speech recognition and onomatopoeia converted by environmental sound recognition are presented on an AR headset. This system achieves multimodal recognition by integrating the results of object detection and environmental sound recognition by Convolutional Neural Network into the final recognition result. We report on experiments showing that multimodal recognition enables highly accurate environmental sound recognition.