2026 年 33 巻 2 号 p. 537-569
Traditional image clustering is constrained by its reliance on single-modal image representations, which limits its ability to capture complex semantic relationships and diminishes the interpretability of clustering results. This study proposes a methodology that initially generates captions directly from images and then integrates the image and caption embeddings to enhance image clustering. This approach leverages automatically generated captions, circumventing the dependency on manually labeled annotations. Experiments on eight datasets demonstrate that multimodal fusion consistently outperforms unimodal baselines: CLIP improves by approximately 5%, whereas BLIP and BLIP2 yield smaller gains. Incorporating image and caption embeddings consistently enhances clustering accuracy. Our comparative analysis further revealed that the effectiveness of textual features varies across models, depending on their multimodal training strategies and encoder architectures. Furthermore, we enhance the interpretability of the clusters by employing language models to generate summaries as explanations for each identified cluster. We evaluate these explanations through quantitative and human assessments. The results demonstrate that the GPT-4 summaries best align with the ground-truth labels and outperform the others in terms of accuracy, coverage, and readability. These findings highlight the effectiveness of sentence-type explanations and the impact of the model architecture and input strategy on explanation quality. Overall, this study establishes a novel image-clustering approach using multimodal representations that combines images and generated captions.
