Speech misperception occurs asymmetrically to a talker ’s facial orientation

Abstract

We conducted four experiments to examine the effect of talker facial orientation on the perception of speech syllables. Similar to the syllables typically used in McGurk effect experiments, the audiovisual stimuli we used in these experiments were the syllables /ba/, /da/, and /ga/, spoken by Japanese talkers. The visual stimuli consisted of a left and right profile view of the talkers, and mirror images of these views. The syllable sound that was uttered was congruently or incongruently combined with the visual stimuli. In Experiment 1, speech misperception of voice significantly increased when the talker ’s face was oriented to the left, indicating that perception occurred asymmetrically, depending on facial orientation. In Experiment 2, we recorded videos of the left and right profiles of the talkers, as well as from the front and three-quarters to the left and right of the talkers. Perceptual error increased again when the talker ’ s face was oriented to the left. Experiment 3 showed that the misperception decreased when only voice stimuli were presented and that the pattern of voice misperception was different from that in Experiment 2. Experiment 4 confirmed that when a face was presented in the absence of voices, and the pattern of speech misperception was also different from that in Experiment 2. These results suggest that speech misperception due to facial orientation asymmetricity occurred not at the level of unimodal, auditory, or visual perceptual processing but at the level of processing involving the integration of audio and visual information.