Conditional image inpainting based on auxiliary information using the energy function of deep generative models

Abstract

In this study, we consider the task of "conditional image inpainting", which aims to restore collapsed images by using the corresponding auxiliary information (e.g., label information). Normally, image inpainting assumes that the process of generating missing or collapsed images is known in advance. In contrast, we consider them unknown and consider inpainting them based on the auxiliary information provided instead. To tackle this task, we first train a conditional model of images given labels based on VAEs. We then estimate an energy function by considering this learned model as a prior. We propose to update images iteratively to minimize this energy function in order to perform conditional image inpainting. Experimental results show that restoration by conditional VAE reconstruction does not consider given labels and that the proposed method can perform conditional image inpainting considering them.