Semantic Similarity Index of Sentences

Abstract

In the operation of machine translation and summarization models, it is possible for the generated sentences to incorrectly state content that is not found in the source text, and this is one of the problems that must be overcome. Many existing automatic evaluation metrics are based on measuring word agreement, making it difficult to take such factual inconsistencies into account. We propose an automatic evaluation metric that focuses on judging the semantic similarity of sentences. When evaluating machine translation and summarization, our method compares an ideal human-created reference sentence with a candidate sentence from a sentence generation system, and outputs a score for the semantic similarity of the sentence pair. The score is obtained by inputting the difference in sentence embedding of the sentence pair obtained from the fine-tuned SentenceBERT to the classifier. The output of image captions is used for evaluation, and the correlation value with human judgments is calculated. It was confirmed that the proposed method has a relatively high correlation with existing automatic evaluation indices. Based on actual operation examples, we also show that the behavior of the proposed method deviates less from human judgments than existing automatic evaluation indices.