Distributed representation learning with syntactic and semantic information based on Hol-CCG and contrastive learning

Abstract

Distributed representations containing not only semantic information but also syntactic information are useful in various downstream tasks of NLP.In this study, we propose a novel training method for distributed representations that include both syntactic and semantic information.The proposed method utilizes Hol-CCG, a syntactic parsing model based on CCG.This model can calculate distributed representations that contain plenty of syntactic information corresponding to each sentence component (i.e., words, phrases, sentence itself).Here, by applying the contrastive learning-based training method to the Hol-CCG, the model is extended to include not only syntactic but also semantic information in the distributed representation it computes.In the experiment, multiple paraphrased expressions were generated for a given sentence, and the similarity of distributed representations calculated by Hol-CCG for both the original and paraphrased sentences was compared.The qualitative evaluation results confirmed that the Hol-CCG trained through the proposed method is capable of evaluating the similarity of sentences that share the same meaning but have different syntactic structures, from both syntactic and semantic perspectives. However, there remain challenges in conducting quantitative evaluation and in the limitations imposed on the sentences eligible for paraphrasing.