Understanding Ambiguous Instructions Using Generative Adversarial Nets for Object Disposal Tasks

Abstract

This paper focuses on a multimodal language understanding method for ``Carry and Place'' tasks with domestic service robots. We address the case of ambiguous instructions, that is when the target area is not specified. For instance ``Put away the milk and cereal.'' is a natural instruction where there is ambiguity on the target area, considering daily life environments. Conventionally, this instruction can be disambiguated from a dialogue system, but at the cost of time and cumbersomeness. Instead, we propose a multimodal approach, where the instructions are disambiguated from the robot state and environment context. We develop MultiModal Classifier Generative Adversarial Network (MMC-GAN) to predict the likelihood of the different target areas considering the robot physical limitation and the target clutter. Our approach, MMC-GAN, significantly improves accuracy compared to baseline methods using instructions only or simple deep neural networks.