Investigating hierarchical grouping in CNNs with data-driven superclass construction

抄録

Recent advancements in CNNs have significantly improved image classification, but the mechanisms behind feature extraction remain an open question. In particular, the relationship between human-defined semantic hierarchies and the visual feature hierarchies learned by CNNs remains unclear, and commonly used superclass labels (e.g., in CIFAR-100) may not align with the internal representation structure of these models. This study examines hierarchical grouping in CNNs using CIFAR-10 and CIFAR-100, focusing on superclass and fine-grained classification by utilizing t-SNE and CKA. The CKA results show that the first half of the layers of CNNs trained with the same dataset but different labels are similar. The results for t-SNE reveal that, in CIFAR-10, models group data by superclass before refining fine-grained categories in deeper layers. In CIFAR-100, the patterns are less consistent, with some superclasses following this trend while others do not, highlighting the limitations of manually defined superclass labels. To address this issue, a method is proposed to construct data-driven, feature-based superclasses using cosine similarity of latent representations and agglomerative hierarchical clustering. This approach leads to more coherent and interpretable clustering in t-SNE visualizations, particularly for VGG16. Furthermore, the proposed framework enables a comparative analysis of architectural differences, revealing that models with sequential feature extraction exhibit clearer hierarchical organization than architectures with residual connections.