Prototype of a Support System for the Hearing-Impaired using Multimodal Environmental Sound Recognition-Based Onomatopoeia Conversion and Speech Recognition

Abstract

This paper focuses on onomatopoeia conversion based on multimodal environmental sound recognition, and reports on a proto-type system that uses this method to support the hearing-impaired. We proposed and prototyped a system using multimodal en-vironmental sound recognition-based onomatopoeia conversion and speech recognition. In this prototype system, text converted by speech recognition and onomatopoeia converted by environmental sound recognition are presented on an AR headset. This system achieves multimodal recognition by integrating the results of object detection and environmental sound recognition by Convolutional Neural Network into the final recognition result. We report on experiments showing that multimodal recognition enables highly accurate environmental sound recognition.