In this paper, we propose using pose estimation extracted from videos to recognize the nurses’ activity when doing endotracheal suctioning. Endotracheal suctioning is a sophisticated method that is very invasive and may accompany risks for patients. As home healthcare becomes more prevalent, there is an urgent need for more certified individuals who can perform endotracheal suctioning. However, quantitative research on nurse care activity recognition in endotracheal suctioning is still limited. To address this issue, our study aims to recognize 9 suctioning activities from video recordings by extracting their pose estimation to classify activities. Because the videos were taken in a real-world environment, there are some obstacles to overcome such as people in the background, and nurses standing out of the frame. Therefore, post-processing needs to be applied after estimating the pose. After pose estimation, these key points are input for Random Forest model to classify activities. Our model achieved an accuracy of 54% and F1-score of 46%.

View full abstract

The detection of a state of non-concentration would allow a system to suggest that workers take breaks to recover their concentration and avoid human errors. Eye gaze information can contribute to the objective analysis of human mental states, such as the concentration state. In this study, we explored a pipeline for constructing machine learning models to recognize the state of concentration using eye-gaze data during reading. The following three stages were included in the evaluation: 1) parameter adjustment, 2) estimation of performance differences in the feature group, and 3) clarification of the limitations of the classification performance. The results show that the classification model performance achieved a maximum F1-score of 0.933, suggesting that Random Forest and a 12-s window size are effective as parameters. This study is expected to contribute to the development of an application for detecting a non-concentration state.

View full abstract

This study presents a comprehensive investigation into stress detection among students, focusing on multiple levels of stress assessment. This research aims to shed light on the complexities of stress experienced in educational settings by utilizing a physiological sensing wristband to capture the multifaceted nature of stress responses. A user study was conducted to calculate the cognitive stress levels of a group of 25 participants by recording physiological signals on an Empatica E4 wristband. Along with the relaxed or non-stressed condition, the study employed a range of simple to complex arithmetic tasks designed to elicit three levels of response: 1) slightly stressed or easy level, 2) stressed or medium level, and 3) highly stressed or hard level. Upon the implementation of multiple deep learning models, FCN, ResNet, and LSTM models demonstrated promising outcomes in accurately categorizing the three different stress levels (easy, medium and hard). The models were trained using KFold and Leave-One-Participant-Out (LOPO) cross-validation techniques. To improve the prediction accuracy of LOPO, a fine-tuning or user-specific data calibration approach was utilized. This approach resulted in significant improvements in accuracy for LOPO, with the FCN model achieving a spike to 60% (F1=0.578), the ResNet model reaching 85% (F1=0.846), and the LSTM model achieving an impressive 91% (F1=0.911) accuracy for three-class classification. Leveraging the insights gained from the prediction outcomes, a prototype application was developed that effectively portrays the dynamic fluctuations in stress levels. This application incorporates a stress meter, allowing users to visually comprehend their stress levels, and it delivers customized alert messages to individuals based on their respective stress levels, ensuring timely support and intervention.

View full abstract

In this study, we estimate the location of caregivers by investigating the signal patterns of detected beacons in different rooms and applying relabeling. Tracking the rooms visited by caregivers is useful to monitor patient and staff assistance. Estimating the visited rooms through the detection of beacons is affected by signal loss resulting in less data. In this study, we investigate the signal patterns in different rooms as the basis for relabeling to use the Received Signal Strength Indicator (RSSI) values from one location as the sample in another location to increase the training set and improve the model accuracy. RSSI-based features are extracted from augmented signal and Random Forest is used for location recognition. Beacon devices were installed in a nursing facility with a frequency setting of 10hz within 5 meters of coverage. The data was collected for 5 days. FonLog, a mobile application is installed in the phone carried by staff in the pocket which is used to gather RSSI readings and log location labels. Training data with increased samples from relabeling based on signal patterns of two rooms achieved an accuracy of 74% which is 5% higher than the performance of the original data.

View full abstract

The abacus (also known as Soroban) is a numerical calculation tool that is traditionally used in East Asian countries. With the advancement of information technologies, the abacus is no longer used as a standard calculation tool. However, abacus learning is garnering global attention due to the secondary skills it can foster, e.g., mental arithmetic ability. Numerical calculation using an abacus requires learning numerical expressions using the beads of the abacus and manipulating beads in multiple ways and in different orders. Due to this complexity, a long period of repeated learning is usually required to acquire the skill of using the abacus. However, the teaching method of the abacus mainly relied on lecturers' observation through finding errors and poor bead manipulations and pointing them out, and there is no other way but to rely on human labor at this moment. In this study, we aim to realize an ICT-based learning support system for arithmetic with a common abacus. This paper proposes a method of estimating input values on an abacus based on image recognition captured by a document camera. Through the evaluation experiments, we have confirmed that the proposed method showed an accuracy of 95.0% in the estimation of 7-digit number input on an abacus. Additionally, this paper will provide discussions to realize the proposed method with other cameras such as wearable camera devices, and to design the coaching system of abacus learning.

View full abstract

In this paper, we evaluate the synchronization method of the robot in virtual and real space in an Augmented Reality (AR) simulator for industrial robots. Human Robot Collaboration (HRC) is working method to perform the same task cooperating with human and robot. The method can combine the advantages of humans and robots to increase working efficiency. It has a lot of potential, but we are unable to implement it in the real world for intimate distance. Therefore, We are developing an HRC simulator using an Mixed Reality (MR) device. The simulator is designed as a HRC simulator that collects various data using VR and MR devices to assist in developing real HRC systems. The simulator has the function of simultaneously operating the robot in the real and virtual worlds. For users to use the simulator without stress, reduces delay between a real robot and a virtual robot is necessary. In recent years, the research on synchronization between virtual and real space has progressed with the spread of digital twins.In this paper, We develop a prototype of an AR simulator and investigate the delay in motion between robots. Then, we design methods to reduce the delay based on the results and evaluate these methods. The prototype consists of a robot moving between two points, and the robot in the virtual space moves to avoid a human hand when it detects it.In this experiment, three patterns are evaluated.The first pattern is a human hand never enters the robot during a motion.The second pattern is a human hand enters the robot only once during a motion.The third pattern is a human hand enters the robot multiple times during a motion.The experiment shows that when the results of the path planning are sent to the ROS and reflected the robot in real space, there is a delay in the motion because the motions are accumulated in a queue.From these results, We design a method to reduce the delay and evaluated it from a position, pose error, and motion delay.

View full abstract

In many developed countries, it is one of the urgent issues to find a way to maintain the health of the elderly people. Specifically, it is desired to monitor always individual elderly living alone or with others, and to notice even a slight lowering of his/her physical ability. A change of the person’s gait is useful for this aim. It is said to be a sign of aging, weakening, and diseases. Especially, it needs to be careful of the risk of dementia when the gait speed falls below a certain standard. Indeed, many studies focus on the gait speed to find some diseases or to detect cognitive decline of the elderly at an early stage. Most of them use digital cameras or accelerometer, but cameras are not always preferable in viewpoint of privacy protection and elderly people may feel unpleasant in forcible attachment of sensors to their bodies. Cameras are also sensitive to the light condition. In this study, therefore, we propose a system in which two analog-output infrared sensors are installed on the ceiling and used for collecting gait data of people passing under the sensors. The record is finally summarized as a monthly report to serve a diagnosis on the degradation of walking ability. The two sensors are used not only to measure the gait speed but also to record the gait waveform for detailed analysis. It is also necessary to know who each pedestrian is. For this goal, we use a finger vein authentication device that locks/unlocks the door at a room. To confirm the validity of our system, we conducted an experiment. The estimated gait speeds had the absolute mean error of 0.025 m/s (standard deviation 0.029). This accuracy seems to be sufficient to study a daily variation. In addition, we succeeded in recording the gait speeds of 12 people for a month. The record not only shows the trend of each individual person over days, but also reveals the variance of gait speed of the same person a day. Indeed, we noticed that some people change the speed largely during a day.

View full abstract

In recent years, IoT devices have become widespread in households, and IoT devices with various functions are sold and used in various situations.However, current IoT devices are black boxes in operation, and there is no way to detect when an IoT device is communicating in a suspicious manner. Therefore, we are aiming to realize a framework called the IoT activity tracker that has a function of access control, which can detect what kind of communication IoT devices are doing and allow only appropriate communication based on it, and a function that enables users to understand the operation status of IoT devices by visualizing what kind of communication IoT devices are doing. To achieve the IoT activity tracker, it is necessary to estimate the function in a few seconds of communication traffic. In this study, we used 8 models of IoT devices to estimate functions at intervals of a few seconds, and estimated 3 functions, including a state in which nothing is being executed, using features at intervals of 1 second. As a result, it was confirmed that the function could be estimated with an accuracy of 83% or higher for 5 of the 8 models, respectively.

View full abstract

An overview of the group activity record is useful to facilitate review for self-reflection and improvement of the activity design. To obtain such a supplemental view for the group activity analysis, this study addresses estimating the unique scales for the target group activity in a data-driven manner, which complementary works with the existing knowledge-based measurement scales. In the proposed method, each ``scene'' in the group activity is represented by frequencies of ``elementary interactions'' occurring in the scene, and the whole group activity is defined by a collection of the scenes. Corresponding them to the relationship between ``document'' and ``word'' in a topic model for text analysis, Latent Dirichilet Allocation (LDA) is employed to estimate latent topics as the unique scales for the target group activity and the temporal transitions of the activity in the topic space. Instead of commonly used low-level features such as visual/motion patterns, the elementary interactions with a little semantics are used as codewords to simplify the interpretation of the estimated unique scales. To explore the feasibility of the proposed method, we performed qualitative analysis of the small group activities using their estimated unique scales and temporal transitions. We confirmed that they are helpful to analyze what activity patterns were formed, how they changed over time during the group activities, and how they were similar/different between two groups.

View full abstract

This paper introduces the identification between safe standup and risky standup activity using a sit-to-stand transition prediction system from 2D pressure sensor data to mitigate the occurrence of unexpected falls. For wheelchair users, the sit-to-stand transition is a vital daily activity requiring considerable physical effort and balance control. Elderly people, especially those with dementia, may experience significant adverse effects if they cannot perform sit-to-stand correctly, which can result in falls and serious injuries. In this regard, an e-textile pressure sensor-based wheelchair opens up possibilities to reduce unexpected falls by tracking behavioral activities, such as sit-to-stand transition. In the laboratory environment, we collect 20 subjects' pressure sensor data from these modified wheelchairs to forecast sit-to-stand activity (e.g.,trying to standup and assistive standup) and other daily activities (e.g., sitting, exercising, and eating). For predicting these activities, we investigated various machine learning techniques, such as ResNet-50, Long short-term memory (LSTM), XGBoost (XGB),Random Forest (RnF), K-Nearest Neighbor (KNN), Support vector machine (SVM). In this study, we also evaluated the performance of various statistical feature sets for 2D pressure sensor data. Overall, the proposed system can potentially improve the safety and quality of life of wheelchair patients by preventing falls and reducing the risk of serious injuries.

View full abstract

In this paper, we propose the activity of daily living (ADL) flow-line evaluation system by performing a series of ADL recognition. First, we propose a estimation of the ADL flow-line by using Gantt Chart as a model; by setting the vertical axis of Gantt Chart as ADL and the horizontal axis as time, we can represent the sequence of ADLs of a resident in a day. Also, by arranging the ADL Gantt-charts of multiple persons, it is also possible to determine who was with whom, when, and where. This allows us to identify the route of infection. we also defined the IAS(Infectivity After Stay)function to represent residual time and integrate into the Gantt-chart. This makes it possible to calculate infections between people who are not in the same place at the same time. The proposed method is implemented by an RFID system, and an algorithm for determining the passage of area boundaries using RSSI and phase is developed to recognize the entry/exit of a place associated with an ADL. RFID antennas are installed at the boundary wall and the phase peak pattern is detected when the transit is occurred. Flow-line is composed by applying the shortest path algorithm to the sequence of transit information. To verify the effectiveness of the proposed method, we conducted experiments in a laboratory environment with six room by 4 scenarios simulating ADLs in an elderly care facility. We evaluated the flow line of one person activity and the flow lines of the 2-person activities of the caregiver and the cared-for person, the infected person and the uninfected person. To evaluate the accuracy of the flow line estimation, we defined a numerical evaluation method that includes the start and end times of movement and stay. We evaluated multiple flow line scenarios and obtained an average accuracy of 79%. We also confirmed that, by taking into account the residual infection time, we can detect, for example, infection at the toilet.

View full abstract

Gaze estimation is an important factor in human activity and behavior recognition. The technology is used in numerous applications such as human-computer interaction, driver monitoring systems, and surveillance. Gaze estimation can be achieved using different technologies such as wearable devices or cameras. Estimating gaze using a webcam can indeed be more accessible and convenient compared to methods that rely on specific hardware like infrared cameras. In this paper, we propose a data acquisition approach for modeling appearance-based webcam gaze estimation. We implement an application to capture gaze points using a common webcam. The application asks to click on the circle displayed on the screen, and whenever the circle is clicked, the face image and the pixel coordinate of the center of the circle are stored. From each of the 17 participants, 50 patterns of face images and pixel coordinate information were collected. The gaze estimation models used were VGG16, ResNet50, EfficientNetB7, and EfficientNetB2. In conclusion, the result of the test set is best for VGG16 (four feature extractors) with an error difference of 2.4 cm. To validate our model, we also applied a leave-one-participant-out cross-validation and found that the participant with the smallest error difference is 2.533 cm and the largest error difference is 4.759 cm. The study contributes to proposing the data collection method, the best prediction model, and discovering the difficulty of prediction occurs with human individual differences for webcam-based gaze estimation.

View full abstract