Automatic Evaluation and Analysis of Image Captioning Models Based on Scene Graphs

Abstract

Image captioning studies rely heavily on automatic evaluation metrics such as BLEU and METEOR, which are based on n-grams. However, these metrics have shown poor correlation with human evaluations, leading to the proposal of alternative metrics such as JaSPICE. JaSPICE has only been validated for a general image captioning task without an error analysis. In this paper, we analyze JaSPICE for a fetching instruction generation task and identify its errors for an image captioning task. We conducted experiments on STAIR Captions and PFN-PIC datasets and JaSPICE outperformed the baseline metrics on the correlation coefficient with human evaluation.