Measure for Evaluating Textual Coherence Based on Document Hierarchical Structures towards Supporting Document Polishing

Abstract

In this paper, we propose a measure to evaluate the global textual coherence taking into account the document hierarchical structures. In general, a technical article is hierarchically structured in a consistent manner, including paragraphs, sections, and chapters. A textual coherence is a feature in that sentences are well-connected and are placed with modularity in terms of a topic. We produce the graph whose nodes and edges represent sentences and similarities between the sentences, respectively. The coherence at a layer of the document hierarchy is calculated by aggregating the current value computed from the partial graph among the document structures at a layer with the coherence value of the lower layers. The coherence of an entire document is obtained by iterating the above computations in a bottom-up manner. To compare our measure with the conventional ones and examine our performance, we have carried out two experiments, the one employing Discrimination and Insertion, and the other using the real technical papers written and revised by students and commented by supervisors. As a result, it has been shown that our measure can surely catch and quantifying the textual coherence considering document hierarchy that is hard for the conventional measures to treat.