Singing Facial Animation Synthesis Using Singing Voice and Musical Information

Abstract

In three-dimensional computer-graphics speech animation, head rotations and facial features other than mouth motions are considered to be equally important as mouth motions. Such motions especially in singing animation is known to affect the impression of the viewers and requires a huge amount of work by hand when creating them. Therefore, there is high demands in creating such animation with available inputs. In this work, we present singing animation synthesis method with the information associated with the song. Using singing voice, musical information and the mouth motion estimated from the singing voice as input, the proposed method can automatically create realistic head motion and expressions around eyes. We have conducted the numerical evaluation as well as subjective evaluation on the naturalness and the synchronization to the singing voice and song to show the effectiveness of our method.