Learning Egg Drilling Task by using Multimodal RSSM with a real robot

Abstract

In this paper, we train a Multimodal Recurrent State-Space Model (MRSSM) for an egg drilling task, analyze the composed state space and control a real robot. One of the methods of biological experiments is to create a cranial window in a rat, and there is an egg task as a mock task. It is difficult to distinguish contact and non-contact states between the egg and the drill in the Egg Task by using only image information. However, if we use the image and audio information, the state space that separates contact and non-contact states can be composed. In the experiments, we analyzed the transitions of the latent states, and the real robot was controlled by MRSSM. The results of the trained MRSSM with images and audio information show different transitions between the contact and non-contact states. In addition, we confirmed that the MRSSM could control a real robot.