Abstract In the research for sign recognition though machine learning, annotation by human effort was thought to be necessary process. The new idea is proposed here to annotate automatically signing using the audio data recorded simultaneously in the video, which would reduce the time and huma effort. The idea includes the concept of the peculiarity of sign language as bi-modality and translanguaging in bilingualism.