Weakly-Supervised Learning for Continuous Sign Language Word Recognition Using DTW-Based Forced Alignment and Isolated Word HMM Adjustment

Abstract

The reduction of the manual work of annotation is an essential part of sign language recognition research. This paper describes one weakly-supervised learning approach for continuous sign language word recognition. The proposed method consists of forced alignment based on dynamic time warping and isolated word hidden markov model adjustment using ‘embedded training’. While the proposed forced alignment only requires one manual annotation for each isolated sign language word, it can generate sufficient quality of the annotation to initialize isolated word hidden markov models. ‘Embedded training’adjusts initial hidden markov models to recognize continuous sign language words using only ordered word labels. The performance of the proposed method is evaluated statistically using a dataset that includes 5,432 isolated sign language word videos and 4,621 continuous sign language word videos. The averaged alignment error of the proposed forced alignment was 4.02 frames. The averaged recognition performances of the initial models were 74.82% and 91.14% in the signer-opened and trial-opened conditions, respectively.Moreover, the averaged recognition performances of the adjusted models were over 65.00% for all conditions.The evaluation shows significant improvements compared to the previous weakly-supervised learning.