Application of Multimodal for Lithology-aware Prompt Engineering in Mineral Processing Systems

Abstract

Coal and gangue are two important factors in the coal mining, and separating them from each other is one of the steps in the coal mining engineering. Because of the transportations and personnel fee, coal mining engineers should focus on separating coal and gangue from each other as much as possible to save the corresponding cost. Traditional coal mining industry is high-risky and labor-intensive, and simultaneously, until now the number of young professional mining engineers is still decreasing. The mentioned challenges are driving the government to transfer the coal mining industry from human-centered (human eyes locating, human hands taking) to automatic-robots-industry-centered (sensor/camera locating, robots arms taking). Up to now, combined with the robot arms, there are several computer vision algorithms (YOLO, RCNN) trained with open-source public dataset, and applied in the practical coal mining projects. During these projects, there are still some tasks that have not been solved already, i.e., to some degree, open-source public dataset cannot cover all the practical conditions. Therefore, discovering an approach of increasing the diversity of the existing dataset is in need. Based on the aforementioned issue, this research proposes to prove the possibility of applying the Generative Artificial Intelligence (AI) as the supplementary of the open-source public dataset (i.e., generating gangue images). Generative AI-based dataset mainly includes 2 patterns, individually: txt2img and img2img. With the assistance of the multimodal, the possibility of applying lithology-aware prompt engineering in generating gangue images has been proved. And the authors have categorized and analyzed the lithology-aware prompts.