Pose initialization method of mixed reality system for inspection using convolutional neural network

Abstract

The Mixed Reality (MR) has become a trend in industrial applications such as inspection and maintenance thanks to the benefit of technological advances in computer vision. Simultaneous Localization And Mapping (SLAM) is a key component of the MR system which augments the CAD model of a target object in the live stream. However, the existing SLAM-based systems rely on a manual handling or a marker-based registration between the model coordinate and the global coordinate. In this paper, we present a non-marker based registration method which automatically performs both the target object detection in the live stream and its initial 3D pose estimation. We exploit two Convolutional Neural Networks (CNNs) to align the CAD model in a global map, and to detect the target object in keyframes of the SLAM system. Since manual preparation of training data is very laborious, we also propose a data argumentation method for the industrial application. The data augmentation method generates a synthesized dataset consisting of pairs of the RGB image and the corresponding camera pose using the object's CAD model. Two CNNs for the object detection in keyframes and the initial pose estimation are trained with the synthesized dataset, respectively. Our result shows that this method can robustly find the target object's initial pose without a dense point cloud or other features detected by hand-crafted descriptors.