Document clustering is one of the useful approaches for macro analysis of the large scale of documents. However it is difficult for an analyst to efficiently detects clusters which contain important information from the results of document clustering. This paper presents a method to support an analysis of social problems from newspaper articles. We define two new measures for each cluster to discover important clusters from a dendrogram generated by hierarchical clustering algorithm. One, called ``Density'', is a measure of relevance among documents in a cluster, and is calculated from the rate of terms shared within a cluster. The other, called ``Centrality'', is a measure of relevance among clusters, and is calculated from the depth of an ancestor node shared by arbitray two clusters in a dendrogram and the number of documents in the clusters. The measures are an extension of the conventional research in the field of co-word analysis in science and technology literature. We carried out experiments to evaluate our method using the Nikkei newspaper articles which describe the organizational hazards caused by Japanese industries. The experimental results showed that our method efficiently provided useful information to detect important clusters from a dendrogram.

View full abstract

In this paper, we propose a method of non-factoid Web question-answering that can uniformly deal with any class of Japanese non-factoid question by using a large number of example Q&A pairs. Instead of preparing classes of questions beforehand, the method retrieves already asked question examples similar to a submitted question from a set of Q&A pairs. Then, instead of preparing clue expressions for the writing style of answers according to each question class beforehand, it dynamically extracts clue expressions from the answer examples corresponding to the retrieved question examples. This clue expression information is combined with topical content information from the question to extract appropriate answer candidates. The score of an answer candidate is measured by the density of submitted question's keywords, words associated with the question and the clue expressions. Note that we utilize the set of Q&A pairs, not to find answers from them, but to obtain clue expressions about the writing style of their answers. The information source for question answering is the Web documents retrieved by using an API of a Web search engine. Experimental results showed that the clue expressions obtained from the set of examples improved the accuracy of answer candidate extraction.

View full abstract