[Paper] Reproducing material-specific appearances of optically complex objects using a view interpolation network

Abstract

In this study, we attempt to reproduce material appearance of objects with various optical characteristics using a free viewpoint image generation network. The network takes RGB and depth images captured at four specific viewpoints as input and generates an image at an intermediate viewpoint inside them. The RGB images are geometrically transformed into images seen from the output viewpoint by image warping. Then, an image with interpolated luminances is generated using a U-Net based image transformation network. We use adversarial loss to generate material-specific appearances rather than obtaining optically correct outputs. In our experiments, we used metal materials that reflect the surrounding environment, glass materials that transmit and refract light, and materials with sub-surface scattering. The results showed that the use of adversarial loss gave better results for all these materials both in LPIPS, an image quality assessment metric that is close to human perception, and evaluations by human participants.