Real-world robot control based on gated contrastive learning of multi-view camera images

Abstract

Intelligent robots that can adapt their behavior according to their environment are highly desirable. In particular, camera images are often required to acquire environmental information. Obtaining images from multiple viewpoints can help robots to deal with disturbances and self-occlusions. However, simply combining image features from multiple viewpoints can result in the loss of important information. To address this issue, we propose a deep learning-based approach that uses a gating mechanism to weigh the features of multi-view images based on the current situation. Specifically, we use contrastive learning to align the features and then calculate the weighted average of latent representations using the gating mechanism. We also introduce data augmentation to simulate occlusions and auxiliary costs for action predictive learning. To evaluate our approach, we conducted a real robot experiment, and the results demonstrated the effectiveness of each component of our proposed method.