モダリティ注意による深層予測学習の解釈性とノイズロバスト性の向上

―日立-早大の共同研究開発事例―

抄録

In order for robots to perform tasks that humans perform, it is necessary to process multimodal information such as vision and force, just like humans. In this study, we propose modality attention by deep predictive learning that can interpret which modal information is used during the task. A hierarchical model consisting of low-level NNs(Neural Networks) that process each modal information individually and a high-level NN that integrates the modal information is used. Furthermore, by weighting each modal information input to the upper NN with learnable weights and inputting it, the modal information used for motion generation is self-adjustable. We verified the effectiveness of the proposed method in the task of inserting furniture parts that require vision and force. It was confirmed that the modality that attracts attention transitions appropriately, and that stable motion can be generated even if noise occurs in the modality that does not pay attention.