Journal of Information Processing
Online ISSN : 1882-6652
ISSN-L : 1882-6652
 
Provide Interpretability of Document Classification by Large Language Models Based on Word Masking
Atsuki TamekuriSaneyasu Yamaguchi
Author information
JOURNAL FREE ACCESS

2024 Volume 32 Pages 466-470

Details
Abstract

Deep neural networks have greatly improved natural language processing and text analysis technologies. In particular, pre-trained large language models have achieved significant improvement. However, it has been argued that they are black boxes and that it is important to provide interpretability. In our previous work, we focused on self-attention and proposed methods for providing and evaluating interpretability. However, the work did not use large language models, and the evaluation method used unusual sentences by deleting words. In this paper, we focus on BERT, which is a popular large language model, and its masking function instead of deleting words. We then show a problem of using this masking function to provide interpretability, which is that the mask token is not neutral for decision. We then propose an evaluation method based on this masking function with training to learn that the mask token is neutral.

Content from these authors
© 2024 by the Information Processing Society of Japan
Previous article Next article
feedback
Top