Development of Gaze Allocation Data Feedback System for Train Drivers

Abstract

This study aimed at developing a gaze allocation data feedback system for conducting driving simulator training using quantitative gaze data of train drivers. The system can preset driving scenes to efficiently instruct trainees within a limited training time. The system can also compare trainees with experts using face direction data and gaze target data graphs to clarify the characteristics of trainee allocation of visual attention. In addition, the system can visualize the gaze allocation data and clearly show the objects that trainees were looking at while driving a simulator. Instructors from a railway company tested the system and gave a positive evaluation of it as a training tool for trainee visual attention.

1. Introduction

1.1 Background and purpose

Train drivers are required to deal with various abnormal events, such as signal problems and obstacles on the track. In order to improve driver skills in dealing with abnormal events, railway operating companies often conduct vocational training sessions using a driving simulator. Endoh [1] proposed a vocational training program to improve train driver skills for dealing with abnormal events promptly and accurately. The training program included a support system for reflection (Fig. 1), in which drivers could review video recordings of their behavior while driving the simulator and reflect on their psychological condition and reactions to abnormal events.

Fig. 1　Example of screen of the support system for reflection

However, for quick detection of abnormal events, the way drivers allocate their visual attention is crucial. In previous studies in the railway field, Yamauchi [2] and Yamauchi et al. [3] argued that evidence-based instruction is effective for railway crew education. It therefore follows that for visual attention allocation, training that uses gaze allocation data to allow trainees to reflect objectively on where they place their visual attention should be effective. In previous studies in the automobile field, Zhang et al. [4] have indicated that training supported by numerical data improved outcomes. They proposed that a driver's gaze allocation, like any other action, can be developed by observing the behavior of a role model, calling this approach “training based on experienced drivers' performance.” They carried out an experiment using a driving simulator to select drivers with the best gaze allocation and obtained video footage of gaze allocation to produce training videos on hazard perception. Their results showed that among novice drivers who received video training, the training based on the experienced drivers' performance helped novice drivers improve their visual hazard perception. According to the previous study, the visual attention of novice drivers can be improved by watching and learning the gaze allocation techniques of experienced drivers.

This study therefore aimed to develop a gaze allocation data feedback system that would allow trainees to reflect on their visual attention immediately after driving a simulator, using an eye-tacking method without driver fatigue. In this study, “feedback” refers to the objective data given to trainees, which is based on the results of their behavior, and related necessary knowledge.

1.2 Literature review of gaze allocation

Several studies have investigated gaze allocation by train drivers. Luke et al. [5] analyzed the gaze allocation of 86 drivers while operating in-service trains to investigate how signal aspect, i.e., the color of a signal, influenced driver gaze allocation and showed that the preceding aspect, the color of the next signal, and signal complexity were important factors that affect driver gaze allocation. Naweed and Balakrishnan [6] examined the tasks and activities of urban passenger-train drivers to understand the nature of visual attention in their driving activities and showed that driving in an urban environment required the mastery of gaze allocation skills. Groeger et al. [7] investigated the gaze allocation of 10 train drivers and showed that approximately 50% of the time spent approaching signals was used to scan the visual scene. The remaining time was spent looking at railway signage and infrastructure, trackside locations, and signals. Although previous studies have investigated ordinary railway-driving situations, few have examined a training method using objective data to detect abnormal events.

It is important to investigate the relationship between gaze allocation and the detection of abnormal events in railway driving. The authors [8] previously examined an effective gaze allocation during coasting at a high speed of approximately 90 km/h and showed that long periods of looking straight far ahead contributed to the detection of abnormal events. Furthermore, the authors [9] investigated an effective gaze allocation while accelerating at a low speed of approximately 30 km/h and showed that drivers who were able to detect abnormal events looked ahead more widely with short glances. The authors [10] showed that the failure to detect abnormal events was due to excessive concentration on looking straight ahead.

There have been many studies in the field of automobile research that have attempted to correlate gaze allocation with driver skill levels. Mourant and Rockwell [11] investigated differences in gaze allocation by novice and experienced drivers and indicated that the novice drivers concentrated their gaze in a smaller area than experienced drivers did. Crundall and Underwood [12] investigated the differences between novice and experienced drivers in gaze allocation under different levels of cognitive load imposed by different road types and found a significant difference between experienced and novice drivers on the dual carriageway. Experienced drivers had a wider search spread along the horizontal axis than novices. Underwood et al. [13] observed how experienced drivers looked at a road scene while watching a video recording taken from a traveling car. The effect of the driving experience was that experienced drivers had increased variance in their gaze as compared with novices. Konstantopoulos et al. [14] focused on experience-related differences in gaze allocation. Their results showed that driving instructors had a broader spread of gaze in the horizontal axes than learner drivers did. According to these previous studies, experience-related differences led to differences in driver gaze allocation, and a wider range of gaze was an effective gaze allocation characteristic adopted by experienced drivers.

In this way, many studies have reported that the gaze allocation of experts differs from that of novices. Regarding the training method to help novices become experts, one of the effective methods is for novices to receive feedback on their own gaze allocation data and reflect on the differences between the characteristics of novices and experts.

2. Development of gaze allocation data feedback system

In order to conduct training using gaze allocation data, we developed a gaze allocation data feedback system that provides trainees with feedback on the gaze allocation data measured while driving a simulator immediately after driving a simulator. The gaze allocation data feedback system consists of a “function of measuring gaze and determining gaze target” and a “function of aggregating and displaying gaze allocation data.”

2.1 Function of measuring gaze and determining gaze target

2.1.1 Eye tracking system

We used EMR ACTUS made by nac Image Technology Inc. to measure gaze. EMR ACTUS used two cameras to capture the images of the head and eyes (Fig. 2). Its features were to measure gaze without contact and calibration. In order to collect gaze data during actual training, it is important to measure gaze without putting a burden on the drivers. Furthermore, in order to collect gaze allocation data similar to that during normal driving, it is also important that the drivers are not aware that their gaze is being measured. In addition, since simulator training time is limited during actual training, it is necessary to collect gaze allocation data efficiently without spending time on calibration. Taking these points into consideration, we used a method that could measure gaze without contact and calibration.

Fig. 2　Imagines of a head and eyes captured (Citation from reference [16])

2.1.2 Determining face direction and measuring gaze

Taking into account the characteristics of the eye mark recorder, the gaze-measuring range was set to be within the front screen of the driving simulator. When driving a train, it is important for the drivers to look not only at the track ahead, but also at the speedometer and timetable in the cab. Therefore, we used image analysis to calculate the amount of head rotation and tilt, determining the face direction (forward, timetable, speedometer, and driver's desk) (Figs. 3 and 4).

Fig. 3　Area of face direction

Fig. 4　Calculating the amount of head rotation and tilt (Citation from reference [16])

In order to determine the threshold for determining face direction, we carried out an experiment to measure face direction while driving a simulator. The participants were four employees of the Railway Technical Research Institute (two males and two females). While driving between stations in the driving simulator, the participants were verbally instructed to look at the timetable, the speedometer, and the driver's desk. The instructions were given twice between each station. From the data of the face angle while gazing at the timetable, the speedometer, and the driver's desk, we set the threshold for determining the direction that the participants were facing. Table 1 shows the participants' attributes and the results of the face direction determination. Regarding the timetable, speedometer, and driver's desk, we confirmed that we could determine the face direction using image analysis. We compared the video footage with the face direction results determined by the image analysis and confirmed that the face direction results determined by the image analysis were generally consistent with the face direction results determined by the video footage.

Table 1　Participant attributes and results of determining face direction

Participants		A	B	C	D
Attributes	Sex	F	M	F	M
	Glasses	w/o	w/	w/o	w/
Determining face direction	Forward (%)	85	94	92	91
	Timetable (%)	5	3	3	4
	Speedometer (%)	7	2	4	4
	Driver's desk (%)	3	1	1	1

After determining whether the participant's face was directed inside or outside the front screen (timetable, speedometer, and driver's desk), we measured the participant's eye movement on the screen and extracted the location of gaze when the face was directed inside the front screen.

2.1.3 Determining gaze target

Since a feature of train driving is that the scene ahead (the location of a traffic signal or a traffic sign etc. on the front screen) is exactly the same when the distance is the same, we measured the coordinates of each object (a traffic signal or a traffic sign, etc.) on the front screen for each distance (in 0.1 m increments), and generated the object area data. We used an OI-Editor made by Emovis Corporation to generate the object area data. After specifying the rectangle of an object on the front screen, the OI-Editor output the data of the continuous rectangular coordinate using object tracking processing while the video was playing. The OI-Editor added frame information from the video to the data of the continuous rectangular coordinate, referring to the distance information in the driving simulator log file, and generated object area data for each distance (Fig. 5). For object tracking, the original video was played in reverse.

Fig. 5　Generating object area data

After the trainees had completed their training drive, we compared the rectangular data of each object with the location of each gaze on the front screen to determine what they were looking at (Fig. 6). We confirmed that it is possible to automatically determine the objects that the trainee was gazing at while driving a simulator by using the pre-generated object area data for each distance.

Fig. 6　Image of determining a gaze target

2.2 Function of aggregating and displaying gaze allocation data

To compare drivers, we aggregated driving data output from the driving simulator (driving speed, powering notch, braking notch), face direction data, and gaze target data based on distance. Figure 7 is a screenshot of the gaze allocation data feedback system. The display of the gaze allocation data feedback system consists of three areas: (1) Area for setting the driving context, (2) Graph display, and (3) Video display.

Fig. 7　Display example of gaze allocation data feedback system

2.2.1 Area for setting the driving context

The area for setting the driving context allows the instructor to select pre-defined driving moments to focus on, according to the purpose of the training. For example, approaching a station or crossing a level crossing. As the time available for actual simulator training is limited, it is difficult to watch through the entire recorded video of a training session. For example, if the total simulator training time is 60 minutes and the simulated driving time is about 40 minutes, this leaves only about 20 minutes to review the gaze allocation data. The driving situation setting allows the instructor to pre-select the driving moments that trainees should review to reflect on their gaze allocation. This contributes to efficient reflection even in limited training time.

2.2.2 Graph display

The graph display collects and displays data on face direction and gaze target, comparing a trainee's data with that of another driver to identify characteristics of the trainee's visual attention. For example, face direction data show whether a trainee tends to look more or less frequently at the front scene or at the speedometer. Additionally, gaze target data show the number of times a trainee looks at traffic lights or level crossings in the scene ahead.

2.2.3 Video display

The video display visualizes gaze data. For example, instructors can instantly switch between a trainee's gaze and another driver's gaze, showing the difference in what they were looking at in the exact same scene.

2.3 Using the gaze allocation data feedback system

We explain how to use the gaze allocation data feedback system with an example that uses the gaze data of an experienced driver as benchmark data.

First, instructors set up the target driving scenarios, such as train delays or receiving operational notifications. An experienced driver then drives the simulator to provide benchmark data. The instructor then talks through this with the experienced driver to find out what they paid attention to when looking ahead while driving. For example, immediately after running through a curve, the experienced driver paid visual attention to an obstruction warning signal for a level crossing. Based on this interview, the instructor identifies key driving points to focus and reflect on during and after the training. For example, the focal driving situation would extend from the location of the obstruction warning signal for a level crossing to the location of the level crossing. The instructor also specifies the objects that drivers should pay visual attention to in each driving situation. For example, departure signals, passengers on platforms, obstruction warning signals for level crossings, level crossings, signs, etc.

Next, the trainees are asked to drive the simulator. Just after the simulator exercise, the instructor then provides feedback on their gaze allocation data, and they reflect on their driving. For example, the gaze target data show that the experienced driver often looked at the obstruction warning signal, whereas the trainee looked at it less often. After that, the instructors provide guidance based on the visualization of the gaze data. Figure 8 shows an example of the visualization of gaze allocation data looking ahead. By comparing what the trainee was looking at with what the experienced driver was looking at in the same scene, it is possible to verify that the trainee was only looking at the traffic lights and the curve ahead, whereas the experienced driver was clearly looking not only at the traffic lights and the curve ahead but also at the obstruction warning signals for the level crossing. In this way, the instructors can confirm the characteristics of visual attention and provide specific guidance to the trainee for improvement using visualized gaze allocation data.

Fig. 8　Example of visualizing the gaze allocation data in a front scenery

3. Evaluation of the effectiveness of the gaze allocation data feedback system

In order to evaluate the effectiveness of the gaze allocation data feedback system, we installed the system on a driving simulator equipped with an eye tracker used by a railway operating company. Figure 9 shows the eye mark recorder installed. Four instructors involved in simulator training tried it out and answered questions about its effectiveness in training (Fig. 10). As a result, we received positive feedback, such as, “This system is a tool that makes it possible to provide instruction about the driver's visual attention that was not possible before,” “This system makes it possible to provide more convincing instruction based on data,” and “This system makes it possible to provide efficient instruction by aggregating data with the distance.”

Fig. 9　Eye mark recorder installed in the experiment

Fig. 10　Trial of the gaze allocation data feedback system

4. Conclusions

In this study, in order to implement simulator training using gaze allocation data, we developed a system called the gaze allocation data feedback system. This system provides trainees with feedback on their gaze allocation data measured during simulated driving immediately after driving a simulator. By using the gaze allocation data feedback system, instructors can specify which pre-defined driving moments from the recordings will form the focus of instruction. Instructors can then invite trainees to reflect on their gaze allocation data efficiently within the limited training time. In addition, with graphs of the face direction data and gaze target data, instructors can understand the characteristics of the trainees' visual attention compared with those of experienced drivers. Furthermore, with videos that visualize gaze allocation data, instructors can clearly show trainees the differences in visual attention between trainees and experienced drivers when they look at exactly the same scene. Instructors involved in simulator training at a railway operating company trialed the system and gave a positive evaluation.

Further study is needed to investigate how this system can be integrated with actual simulator training and how feedback on gaze allocation data for each training scenario can be provided.

This study was published in RTRI Report (in Japanese) [15] in 2024. Part of this paper was published in reference [16].

Acknowledgment

We would like to express our gratitude to the Hokkaido Railway Company and West Japan Railway Company for their considerable cooperation in the simulator training.

References

• [1]  Endoh, H. and Omino, K., “Practical System for Implementing Vocational Training Program for Improving Train Driver Skills for Coping with Abnormal Situations,” Quarterly Report of RTRI, Vol. 54, No. 4, pp. 237-242, 2013.
• [2]  Yamauchi, K., “Evaluation and Promotion of Railways Employee Awareness about Observing Rules on Making Passenger Announcements,” Quarterly Report of RTRI, Vol. 53, No. 4, pp. 241-246, 2012.
• [3]  Yamauchi, K., Kikuchi, F. and Fujinami, K., “Guidelines for Making Emergency Announcements Taking into Consideration the Psychological State of Passengers,” Quarterly Report of RTRI, Vol. 58, No. 2, pp. 153-158, 2017.
• [4]  Zhang, W., Wang, Y., Feng, Z., Zhu, S., Cui, J., Hao, W. and Wang, C., “A method to improve the hazard perception of young novice drivers based on Bandura's observational learning theory: Supplement to expert commentary training,” Transportation Research Part F, Vol. 85, pp. 133-149, 2022.
• [5]  Luke, T., Brook-Carter, N., Parkes, A. M., Grimes, E. and Mills, A., “An investigation of train driver visual strategies,” Cognition, Technology and Work, Vol. 8, pp. 15-29, 2006.
• [6]  Naweed, A. and Balakrishnan, G., “Understanding the visual skills and strategies of train drivers in the urban rail environment,” Work, Vol. 47, pp. 339-352, 2014.
• [7]  Groeger, J. A., Bradshaw, M. F., Everatt, J., Merat, N. and Field, D., “Pilot study of train drivers' eye-movements,” University of Survey Technical Report for Rail Safety and Standards Board, London, 2003.
• [8]  Suzuki, D., Yamauchi, K. and Matsuura, S., “Identifying the effects of visual searching by railway drivers upon the recognition of extraordinary events,” presented at the 20th Congress of the International Ergonomics Association, Florence, Italy, August 26-30, 2018, pp. 165-173.
• [9]  Suzuki, D., Yamauchi, K. and Matsuura, S., “Effective Visual Behavior of Railway Drivers for Recognition of Extraordinary Events,” Quarterly Report of RTRI, Vol. 60, No. 4, pp. 286-291, 2019.
• [10]  Suzuki, D., Koike, T. and Kawarai, R., “Identifying effective visual searching for the recognition of extraordinary events during the acceleration process of railway driving,” presented at the 22nd Congress of the International Ergonomics Association, Jeju, Korea, August 25-29, 2024.
• [11]  Mourant, R. R. and Rockwell, T. H., “Strategies of visual search by novice and experimental drivers,” Human Factors, Vol. 14, pp. 325-335, 1972.
• [12]  Crundall, D. E. and Underwood, G., “Effects of experience and processing demands on visual information acquisition in drivers,” Ergonomics, Vol. 41, pp. 448-458, 1998.
• [13]  Underwood, G., Chapman, P., Bowden, K. and Crundall, D., “Visual search while driving: Skill and awareness during inspection of the scene,” Transportation Research Part F, Vol. 5, pp. 87-97, 2002.
• [14]  Konstantopoulos, P., Chapman, P. and Crundall, D., “Driver's visual attention as a function of driving experience and visibility. Using a driving simulator to explore drivers' eye movements in day, night and rain driving,” Accident; Analysis and Prevention, Vol. 42, pp. 827-834, 2010.
• [15]  Suzuki, D., Kikuchi, F. and Koike, T., “Development of a Gaze Distribution Data Feedback System for Train Drivers,” RTRI Report, Vol. 38, No. 11, pp. 71-77, 2024 (in Japanese).
• [16]  Suzuki, D., Matsuura, S., Koike, T. and Matsuu, K., “Gaze Data Feedback System on Railway Driving Simulator,” The Japanese Journal of Ergonomics, Vol. 55, Supplement, 1B1-5, 2019 (in Japanese).

Authors

	Daisuke SUZUKI, Ph.D. Senior Researcher, Ergonomics Laboratory, Human Science Division Research Areas: Safety Ergonomics, Human Factors
	Fumitoshi KIKUCHI, Ph.D. Senior Researcher, Ergonomics Laboratory, Human Science Division Research Areas: Social Psychology, Emotional Psychology
	Takaharu KOIKE Emovis Corporation Research Areas: Eye Tracking, Gaze Analysis