Multimodal Behavior in E-Learning Scenes: An Analysis of the Processes by which Learners Interpret Lecture Content

Abstract

This paper analyzes the multimodal behavior of learners in e-learning situations. In the two surveys, three multimodal actions were analyzed: nodding, gazing, and note-taking. Nodding was seen (1) while interpreting content, and (2) when interpretation was difficult. Thus, nodding was found to appear according to the progress of interpretation, without assuming others. In regard to gazing, participants were found to (1) pay more attention to the media used, and (2) pay more attention to the screen when new or unexpected information appeared. For (1), when participants paid attention to the media used, they often looked at the handouts and actively took notes. While for (2), when new information appeared, the participants looked at the information source not in their own domain, and when unexpected information appeared, being surprised or more alert, they visually followed the information regardless of the mode. In the case of note-taking, (1) leaving the video to play without pausing, (2) pausing the video and note-taking, and (3) rewinding the video after note-taking, were observed. In all cases, the complex process of multiple multimodal actions being performed simultaneously was observed.