Hybrid Caption Including Formula or Figure for Deaf and Hard-of-Hearing Students

Abstract

Captions, which are used as a means of information support for deaf and hard-of-hearing students, are usually presented only in text form. Therefore, when mathematical equations and figures are frequently used in a classroom, the captions will show several demonstrative words such as “this equation” or “that figure.” As there is a delay between the teacher’s utterance and the display of the captions sometimes, it is difficult for users to grasp the target of these demonstrative words accurately. In this study, we prepared hybrid captions with mathematical equations and figures and verified their effectiveness via a comparison with conventional text-only captions. The results suggested that the hybrid captions were at least as effective as the conventional captions at helping the students understand the lesson contents. A subjective evaluation with a questionnaire survey also showed that the experimental participants found the hybrid captions to be acceptable without any discomfort. Furthermore, there was no difference in the number of eye movements of the participants during the experiment, suggesting that the physical load was similar for both types of captions.