Development of Structural Learning Methods for Language Models Using Algebraic Statistical Approaches

Abstract

Algebraic statistics, a fusion of algebraic geometry and statistics, represents probabilistic models as low-dimensional structures in high-dimensional spaces and captures their structures through ideals of polynomial rings. From this perspective, it is expected to naturally model complex constraints such as conditional independence and hierarchical structures in linguistic data. This study formulates probabilistic models of language as joint probability distributions of words composing sentences and applies algebraic statistical approaches. By characterizing the models through vanishing ideals that constrain the probabilistic structures, the study develops methods to select models that fit the data. Experiments using synthetic data demonstrated that the proposed method achieved performance comparable to information criteria and directly extracted the structural characteristics of the probabilistic model.