Compositional Plan Vectors for Instruction Following

Abstract

Instruction following is one of the most important problems to operate a robot in human space. However, it is still an open problem that a robot can not accomplish complex tasks such as housework. Since a task consists of multiple skills, if a representation of a task that expresses this compositionality can be obtained, the robot may be able to accomplish such a complex task. To this end, recently Compositional Plan Vectors (CPVs) were proposed and achieved high task success rate in complex tasks that consist of many skills. However, previous CPVs can not be applied to instruction following because the observation at the goal state is not unique. In this work, following the CPVs, we proposed the method to obtain the compositional task representation in instruction following. Experimentally, we show that our method can improve the task success rate.