Conceptual Knowledge in the Pretrained Japanese BERT Model

Abstract

In this paper, we undertake experiments to identify neurons that hold knowledge about concepts in the Feed-Forward Network (FFN) layers of a Japanese BERT model. Specifically, we identify important weight parameters (neurons) within the pre-trained Japanese BERT model for MLM tasks, and confirm their locations and significance through comparative experiments. Additionally, we compare our results with prior studies using a pre-trained English BERT model, demonstrating differences in experimental outcomes under specific conditions.