Surgical Action Recognition Model Based on Temporal Pose Feature Matrix with an Attention Mechanism

Abstract

The task of exchanging surgical instruments between the surgeon and scrub nurse has been targeted for automation using a scrub nurse robot (SNR) [1,2,3,4,5]. To make the SNR system practical, real-time recognition of the surgeon's intentions is crucial. To achieve this, a classification model called Temporal Pose Feature Convolutional Neural Network (TPF CNN) [6,7] has been developed to recognize surgical procedures based on body movements extracted from skeleton data. In this paper, we propose a new model and report on two modifications aimed at the practical application of the system. Specifically, we introduce the TPF Attention Network (TPF AN), a novel model incorporating a self-attention mechanism [12] that is better suited for processing sequential data. The first modification involves adding a class to recognize when the surgeon is not performing a surgical action. This addition enables continuous inference throughout the entire surgical procedure, from start to finish. The second modification involves evaluating the system using the Leave-Person-Out (LPO) framework. Unlike the Leave-Videos-Out (LVO) framework used in prior research, which included at least one video from each surgeon in the training set, the LPO framework uses all videos from one surgeon for testing while the remaining surgeons' videos are used for training. This ensures a robust evaluation of the model's performance when encountering surgeons not included in the training data. Experimental results show that the proposed model improvements led to a 2.0-point increase in accuracy compared to conventional methods.