Deep Reinforcement Learning for Sensorless Speed Control of Rotary Ultrasonic Motors

Abstract

The use of external sensors can guarantee higher stability and reduced tracking errors on the expense of increased overall system’s complexity and cost. For speed control of ultrasonic motors, an isolated electrode can be embedded within the piezo ring instead of direct speed measurement. The vibration amplitude and rotor speed can be estimated directly through the feedback voltage (FBV) signal. The speed/FBV relationship can be fitted using a linear regressor, or a nonlinear function approximator, and then further integrated with other controllers. However, this complicates controller design due to the decoupling of the speed estimator and the controller. Another control approach that simulatenously perfroms estimation and control is Deep Reinforcement Learning (DRL) which directly learns a model-free optimal nonlinear control policy by iterative trial and error. To apply DRL to sensorless speed control of USM, an NN-based policy learns the mapping from an input state (A Markovian state includes the driving frequency, motor temperature, target speed, and FBV signal) to an optimal action (Driving frequency update step) by maximizing some reward function (a function of the speed error and the control effort). Using the Soft Actor-Critic (SAC) algorithm, optimal sensorless speed tracking was realized both in simulation and experiment.