Few-shot Concept Erasing from Text-to-Image Diffusion Models

Abstract

Text-to-Image Diffusion Models trained on vast amount of web data can generate high-quality images under various conditions. However, they may also generate inappropriate images. To address this issue, methods have been researched for removing specific concepts from the pretrained models. In this paper, we propose a novel approach that involves fine-tuning the text encoder instead of relying on the conventional U-Net. This method allows for the removal of concepts with a couple of real images without compromising the model's generative ability (image fidelity). Our experiments confirmed that the specified real-world concepts are less likely to be generated. Furthermore, while previous methods required human intervention to control how the target concept changes internally within the model, our proposed method yields results suggesting that it leverages internal knowledge or image knowledge within the model.