Generate AI texture Using the Controlnet model of stable diffusion to create stylized materials in Blender

Abstract

This work is a media art piece centered around stylized materials. With the advancement of AI painting technology, texture creation has seen a surge in possibilities and efficiency. Utilizing UV projection from the camera in 3D software opens up various methods for stylization. The process begins with modeling in Blender, where information such as depth and wireframes is extracted from objects within the scene. Subsequently, the Controlnet model of AI painting is used to generate corresponding textures. Camera projected UV mapping is carried out in Blender's Eevee renderer, followed by adjustments and hand drawn details in Photoshop. The modified textures are connected as base colors to a cartoon shader, resulting in a Celluloid (Cel) style material.