A Novel Teleoperation Approach Based on MediaPipe and LSTM

Abstract

In the field of robotics, teleoperation technology is of great significance, which facilitates efficient, safe, and sustainable solutions for human’s work and life. This paper proposes a novel teleoperation approach for mobile manipulators that allows operators to control this robotic system by using one hand. We integrate a hand skeleton detection technology named MediaPipe Hands with the RGB-D camera to obtain 3D keypoint coordinates more accurately. After that, we analyze the various features of our hand in space to form corresponding teleoperation commands. To enable the utilization of particular gestures for transitioning between controlled objects, we introduce a particular hand gesture recognition model leveraging Long Short-Term Memory (LSTM) architecture, which has achieved 100% recognition accuracy for three types of gestures. Ultimately, we perform practical experiments to validate the effectiveness of our approach.