Text Classification using Similarity of Tree Sources Estimated from Bayes Coding Algorithm

Abstract

In this paper, we propose a method of text classification using a Bayes coding algorithm, one of the efficient data compression methods. The Bayes coding algorithm gives the Bayes optimal data compression over the tree source model class. When data is compressed by the Bayes coding algorithm, the probability structure of information sources is implicitly estimated from the compressed data. Therefore, we can expect that the implicit estimation of data compression can be utilized for other purposes, especially for the document classification problem. As for the document classification using data compression methods, ZIP format and context tree weighting methods have been proposed. However, these methods do not have Bayes optimal compression and use the compression ratio as a similarity measure between documents for classification. In the Bayes coding algorithm, a weighted mixture tree given by the compression phase can be used for estimated probability structure. Tree source is a class of Markov sources and it is possible to measure the divergence between the tree sources with the same structure. However, the Bayes coding algorithm outputs different tree structures based on the data sequence to be compressed. Since the tree structures derived from documents are different from each other, it is difficult to measure the divergence between them just as it is. This paper proposes a new method to change the structures of weighted mixture trees into the same tree structure to be able to measure the divergence. Using the divergence between trees estimated by documents, the documents can be classified. Moreover, the effectiveness of the proposed method is clarified via a simulation experiment for the document classification with natural data.