Automatic Stopword Generation Based on Attention for Document Classification Using Neural Networks

Abstract

Stopwords are generally used to improve the accuracy of document classification and retrieval. We believe that setting appropriate stopwords improves classification accuracy. However, in our preliminary experiments, in document classification tasks using BERT, existing stopword lists are not effective for improving classification accuracy. To solve this problem, we construct a method for generating stopwords using the attention mechanism of the classifiers. In this method, words with high attention in misclassified input documents and low attention in correctly classified documents are treated as stopwords. The system probabilistically removes stopwords. The system automatically sets the probability of each word in input documents being a stopword when it builds the classification model. We conduct experiments to confirm effectiveness of our stopword generation method. Our experimental results show that there are cases using stopwords generated by our method that improve the classification accuracy. Three of the six classification tasks tested in this study show significant differences in accuracy improvement.