Performance evaluation and internal state analysis of RCNN model in sign language classification

Abstract

Sign language is a language used among hearing-impaired people. However, it is not common in our society and not many people can understand sign language. Developing a sign language translator is a big challenge for artificial intelligence. The purpose of this study is to classify sign language words in video as the first step toward sign language translation using deep learning. In sign language, not only hand shape and hand trajectory but also non-manual signals such as facial expression and nodding are important to understand meaning. A sign language translator needs to take into account the whole image of speakers. In this study, we applied the RCNN model which can use the information of the whole image and classified the sign language words.We also examined how input and model structure effect classification accuracy.