A 3D object erasure and replacement using AR and general object recognition

Abstract

In this study, we developed a system that replaces real objects with virtual objects of the same category and displays them. By combining general object recognition and inpainting into AR, real objects are visually erased and only virtual objects of the same category as real objects are displayed. As a result, the real object appears to be replaced by a virtual object of the same category on the screen. The system obtains the category and area of the object to be removed by general object recognition, sets an image that the object is erased by Inpainting as the AR background, and places virtual objects of the same category as the recognized category. In some scenes, real objects block the plane where the virtual object is placed, so the depth image is modified to recreate the hidden plane, and the virtual object is placed using the repaired depth image. The system works with a single smartphone equipped with an RGB camera.