Prototyping of a real-time sound effect tool for live streaming with CG avatar

Abstract

In this study, we created a prototype of a tool that plays sound effects in real time in response to the actions of a CG avatar. We analyzed three VTuber live streams focusing on the actions and confirmed seven types of repeated actions and characteristic actions. The analysis of one of the movements, a small wave of the left hand, showed that the z-axis rotation occurred with a fixed period, and the angular velocity changed periodically around 0. Based on the results, we created a prototype that detects the motion and plays a sound effect using VIVE Pro 2. Using the prototype, we confirmed that the sound effect was played back along with the movement. We also ported some of the functions to HoloLens 2 and discussed them.