We propose a method for constructing an interpersonal interaction system using a real image-based avatar. Human-computer interaction is important when we communicate with computers. As a medium of an interpersonal interaction, communication robots are used commonly in the real world and CG avatar is used in the virtual world. On behalf of the communication robots, android robots that have a similar appearance to an actual person can effectively transfer the human presence. On the other hand, CG avatar has advantages of a cost and installation space as compared to communication robots. When we use a CG avatar for a communication, it is important to increase the reality of a speaker. In this paper, we construct a prototype of an interpersonal interaction system using a real image-based avatar. By generating the avatar from captured images, we can have a high realistic sensation similar to the interpersonal communication system such as a video conference system. In the proposed method, additionally, the movement of the avatar is smoothed by generating interpolated images. The interpolated images are generated by approximating avatar's each body part as an ellipsoid, using user's posture information. To verify the validity of the system, we have conducted the experiments of subjective evaluation.