For practical holographic video display, generating the holographic fringe as fast as possible is important. In this paper, we report how we used the image hologram for fast calculation and the rainbow hologram for full-color images. In both image and rainbow holograms, image points are located close to the hologram and the diffraction angle is limited. This makes the required computation area much smaller than that of the entire hologram. Therefore, we obtained vastly improved computational speed. From the experimental results with an image hologram with 1.47 million pixels, the speed of ten frame per second was obtained for an object that consists of about 7,000 points. In the rainbow hologram with 1.44 million pixels, the speed of ten frame per second was obtained with an object that consists of about 17,000 points. These results show that the proposed method is quite effective for interactive holographic video displays.