Switching Head-Tail Funnel UNITER: Multimodal Instruction Comprehension for Object Manipulation Tasks

Abstract

This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. We propose Switching Head-Tail Funnel UNITER, which solves the task by predicting the target object and the destination individually using a single model. We conduct physical experiments in which a DSR delivers standardized everyday objects in a standardized domestic environment as requested by instructions with referring expressions. The experimental results show that our method outperforms the baseline method in terms of language comprehension accuracy and the object grasping and placing actions are achieved with success rates of more than 90%.