Real-time Moving Image Synthesis of Human Facial Expressions Based on Hand Shape Recognition

Abstract

To realize a human-like software-robot for better human interface system, we must integrate the technologies of computer graphics, computer vision, speech recognition, voice synthesis and others. We have named this type of the software robot to be served in an advanced human interface system as “Visual Software Agent (VSA)”, and begun our research project for its realization. In the VSA, it is very important to realize real-time processing of several tasks to achieve the real-time response of the software robot. In this paper, we describe a real-time image recognition and image synthesis system, “TN-VIT (Transputer Network with Visual Interface for Transputers)”, as a first step to build the “VSA”. In this prototype system, a human face (agent) is shown on the CRT, and he/she gazes at operators and changes his/her facial expressions in response to operator's hand shape in real-time. Therefore, the system recognizes hand shape of an operator, then synthesizes huaman facial expressions in real-time. The VIT is our original Transputer board with functional 32-bit parallel visual data bus. The maximum visual data transfer speed of our VIT is 100 Mbyte per second. The TN-VIT system consists of 16 VITs, 16 normal Tranputers, 1 personal computer, 1 CCD camera, 1 A/D board and 1 D/A board. The 16 VITs are connected as a ring-type bus, and 2 of them are used for image recognition and the rest of them are used for image synthesis. Experiments show that the TN-VIT system can synthesize natural moving human face images and his/her emotional expressions at 8 frames per second in response to the hand shape of an operator. Details of the hardware and software are described in this paper along with experimental results including some samples of original image and synthesized images.