In recent years, research on autonomous ships has been conducted worldwide. The authors have also developed an automatic collision avoidance system, and have reached practical use level as elemental technology through actual ship experiments. However, there has not been enough research and discussion on the evaluation method for the purpose of certification of this automatic collision avoidance system. In this paper, we propose a method to evaluate the safety of automatic collision avoidance systems for the purpose of class certification in the future. The proposed evaluation method is based on the assumption that an automatic collision avoidance system for unmanned vessels (remote operation) does not cause anxiety on manned conventional vessels and basically follows the current COLERGs. In addition, according to the proposed evaluation method, evaluations were conducted for the purpose of safety certification using the automatic collision avoidance system developed by the authors as an example.