A Transfer Learning Method for the Multimodal Language Understanding Based on Dual ProtoNCE

Abstract

We focus on the task of identifying target objects in domestic environments according to free-form natural language instructions. In this study, we propose a novel transfer learning approach for multimodal language understanding, Prototypical Contrastive Transfer Learning (PCTL) which uses a new contrastive loss, Dual ProtoNCE. Our experiment demonstrated that PCTL outperformed existing methods.