Intelligent Sound-Sensing System Adaptable with Visual Information

Sensor Fusion of Audio and Visual Sensors

Abstract

A new method for adapting an acoustical sensing system to sounds by visual sensor data is proposed.
The acoustical processing part of the system is an intelligent sound sensing system, which can extract a target sound from other disturbance sounds if the envelope of the target signal is estimated.
The spatial filter technique is utilized for a visual image processing part of the system. A vertical-stripe spatial filter estimates the envelope of the sounds from sound sources swinging horizontally. A horizontal-stripe spatial filter estimates the envelope of the sounds from sound sources shaking vertically. Using spatial filter technique, the system can distinguish the target sound from other disturbance sounds by its property of moving. Furthermore, using the estimator of envelope of a disturbance sound also, the system can adapt to the sounds whose envelopes have any correlations each other.
Theoretical analysis and experiments demonstrated adaptability of a linear filter for acoustical signal paocessing only by visual information.