[Paper] Real-Time Free-Viewpoint Image Synthesis System Using Time Varying Projection

Abstract

Given video inputs from multi-view cameras, our goal is to synthesize high quality free-viewpoint images in real-time. To this end, fast and accurate depth estimation from multi-view images is indispensable. Triangulation among the cameras often produces erroneous depth information, especially for textureless regions, resulting in low-quality synthesized views. To improve the quality of depth estimation while keeping fast computational speed, we combined active illuminations using a DLP projector with passive triangulation using multi-view images. The projector casts spatially incoherent patterns to the scene and makes uniform regions texture-rich so that triangulations for those regions can be greatly improved. Moreover, making the illuminations time-varying, we can further stabilize depth estimation using spatiotemporal matching, and remove the artificial patterns from the synthesized virtual views by averaging successive time frames. Our system consists of 16 video cameras synchronized with the DLP projector, and can synthesize high quality free-viewpoint image in about 10 fps.