Novel view synthesis for 360 degree images with NeRF

Abstract

In recent years, neural radiance fields (NeRF) has dramatically improved the performance of novel view synthesis using perspective images. However, no study has addressed the problem of applying NeRF to 360° images in ERP format. For 360° images in ERP format, NeRF’s general ray sampling strategy is ineffective due to the characteristics of spatial distortion in high latitude regions and a 360° wide viewing angle. Therefore, we propose two non-uniform ray sampling schemes, distortion-aware ray sampling and content-aware ray sampling, to make NeRF applicable to 360° images. We created an evaluation dataset using Replica and SceneCity models of indoor and outdoor scenes, respectively. In experiments, we show that our proposed method successfully builds NeRF for 360° images in terms of both accuracy and efficiency. Furthermore, we showed that NeRF++ combined with our proposal accurately synthesizes arbitrary views from a set of 360° images of real scenes.