Multi-modal NewtonianVAE: High-precision reaching method for autonomous suturing

Abstract

This study focuses on NewtonianVAE, a world model that can learn a proportionally controllable latent space. To achieve precise control in a physical world, it is necessary to construct a latent space of NewtonianVAE representing a physical world from multi-modal observations. However, learning from multi-modal observations using NewtonianVAE has not been studied. To address this issue, we discuss methods for learning multi-modal observations using NewtonianVAE. In this paper, we propose Multi-modal NewtonianVAE (MNVAE), which uses Mixture-of-Products-of-Experts (MoPoE) to integrate multi-modal observations. MNVAE learns a latent space representing a physical environment and it has the potential for precise control in a physical world.