Softness Estimation Based on Images of Pushing Action Using Deep Learning

Abstract

In this research, we propose a method that estimates the softness of an object based on the motion images of a hand and a forearm when a target object is pushed with a stick. The softness of the object is estimated from those images by using deep learning. For the motion recognition, we capture a series of RGB-D images with a depth camera. A subject pushes objects of different softness with a stick for collecting motion images for learning. Then the captured images are learned through Convolutional Neural Network and their characteristics are parameterized appropriately to achieve the softness estimation system. The results of softness estimation show that root mean square error of the estimated value of non-learned softness scores within 5 points in durometer hardness. It means that pushing motions of human beings include tactile information that leads to estimate the target object softness and our system can recognize it accurately. We also confirmed that using all the 3 types of images (RGB-image, depth image and Canny edge image) as the input results in the highest accuracy for both personalized and generalized networks.