Iterative Self-Improvement of Vision Language Models Explaining Decision Rationales

Abstract

Image scoring is a crucial task with many practical applications. To trust a model's judgement, understanding its rationale is essential. This paper proposes a novel training method for Vision Language Models (VLMs) to generate not only image scores but also corresponding justifications by natural languages. Leveraging only an image scoring dataset and an instruction-tuned VLM, our method enables self-training, employing the VLM's generated text without relying on external data or models. In addition, we introduce a simple method for creating a dataset designed to improve the consistency between predicted scores and their textual explanations. Through iterative training of the model with Direct Preference Optimization on two distinct datasets and merging them, we can improve both scoring accuracy and the coherence of generated explanations.