Online Acquisition of Japanese Unknown Morphemes using Morphological Constraints

Abstract

To solve the unknown morpheme problem in Japanese morphological analysis, we propose a novel framework of online unknown morpheme acquisition and its implementation. In online unknown morpheme acquisition, an unknown morpheme acquirer, which works in concert with the morphological analyzer, detects unknown morphemes from each segmented and POS-tagged sentence, enumerates its possible interpretations, and selects the best candidate. In enumeration, morphological constraints of the Japanese language are utilized, and selection is done by comparing multiple examples kept in storage. When the number of examples being compared is large enough for disambiguation, the acquirer directly updates the dictionary of the analyzer, and the acquired morpheme will be used in subsequent analysis. Experiments show that unknown morphemes are acquired from relatively small numbers of examples with high accuracy and improve the quality of morphological analysis.