Improvement in Domain Specific Word Segmentation by Symbol Grounding

Abstract

We propose a novel framework for improving a word segmenter using information acquired from symbol grounding. The framework uses a dataset consisting of pairs of non-textual information and a commentary. We generate a pseudo-stochastically segmented corpus from the commentaries, and then build a neural network to predict relationships between non-textual information and the words. We generate a domain specific term dictionary by using the neural network for word segmenter. We applied our method to game records of Japanese chess with commentaries. The experimental results show that the accuracy of a word segmenter can be improved by incorporating the generated dictionary.