The generation of an image of objects from somatosensory information during haptic exploration.

Abstract

Visual occlusions caused by the environment or by the robot itself can be a problem for object recognition during manipulation by a robot hand. Under such conditions, tactile and somatosensory information are useful for object recognition during manipulation. Humans can visualize the appearance of invisible objects from only the somatosensory information provided by their hands. In this paper, we propose a method to generate an image of an invisible object’s posture from the joint angles and touch information provided by robot fingers while touching the object. We show that the object’s posture can be estimated from the time-series of the joint angles of the robot hand via regression analysis. In addition, conditional generative adversarial networks can generate an image to show the appearance of the invisible objects from their estimated postures. Our approach enables user-friendly visualization of somatosensory information in remote control applications.