Endoscopic Sinus Surgery (ESS) requires precise operation of endoscope and surgical instruments in a narrow space adjacent to the eyes and brain. Therefore, surgeons need high surgical skills. In the previous research, process analysis methods were developed which enable detailed evaluation of surgeon skills and detection abnormalities per surgical phase. These methods need manual annotation of surgical situation. However, a method for automatic recognition of surgical situation of ESS is required for system automation. Several methods regarding surgical situation recognition have been reported in other surgery. Particularly, the method using surgical images has achieved high accuracy. Since ESS operates with the right hand while operating the endoscope with the left hand, there are many blurring and screen defects of the image, and it is conceivable that high accuracy cannot be obtained from the image alone.

In this research, we proposed the first automatic situation recognition method using endoscopic images and surgical navigation information in ESS. From experimental results using clinical data, total recognition result of the two types of surgical situations with different granularity was 62.9%. We suggested that the recognition of the surgical situation which is difficult only with images by the endoscope position/orientation data from the navigation information.