Integrating vision and tactility based on interactive perception for estimating opening of a bag

Abstract

Plastic bags are challenging objects for robot manipulation due to transparency and deformability. This paper proposes a learning approach for a robot to insert its hand into a bag-shaped object based on visual and tactile sensing. The basic idea is to utilize probing action that allows to acquire rich information about the object even with a simple tactile sensor. The structure of the object is estimated by unsupervised learning with contact and reachability information. The result is transferred to visual recognition as self-supervised learning. Based on the unsupervised learning result, the robot can verify whether the hand truly reached the interior of the bag by additional probing actions. The proposed method was evaluated experimentally by a robot hand with a simple tactile sensor.