In this paper, we improve nursing activity recognition in gastrostomy tube feeding (GTF) with temporal variations and sequential errors by integrating activity context to Large Language Model (LLM) for guided feature selection and post-processing. GTF is a delicate nursing procedure that allows direct stomach access in children for supplemental feeding or medication, but it is underrepresented in datasets, posing challenges for accurate detection. Manual feature engineering may overlook subtle but important motion cues, particularly in opening and closing the gastrostomy cover, where changes are minimal and localized to the hands. Additionally, sequence inconsistencies and missed activities limit the effectiveness of pose estimation methods in healthcare. Leveraging the contextual adaptability of LLMs, we generate new features suggested by the language model, combining them with hand-crafted features to optimize the model. For post-processing, a sliding window smoothing method based on majority voting is applied. To mitigate duration-based discrepancies, a priority handling is incorporated for short-duration activities to pre- serve their recognition accuracy while addressing repeated labels caused by long-duration actions. Particularly, we applied activity recognition to our unique GTF dataset collected from recorded video of two nurses, two students, and two staff members for three days with 17 labeled activities. Keypoints are extracted using YOLO11. Compared to the baseline, the application of LLM to GTF nurse activity recognition with pose estimation improved the Random Forest performance of F1-score from 54% to 57%. Additionally, incorporating the sliding window smoothing approach based on majority voting with short-term action priority, resulted in a 3% further increase.

View full abstract

In this study, we propose to optimize temporal parameters with pose estimation data of simulated abnormal activities of developmentally disabled individuals by incorporating behavior context to Large Language Models (LLMs). Facilities for the developmentally disabled face the challenge of detecting abnormal behaviors because of limited staff and the difficulty of spotting subtle movements. Traditional methods often struggle to identify these behaviors because abnormal actions are irregular and unpredictable, leading to frequent misses or misclassifications. The main contributions of this work is the creation of a unique dataset with labeled abnormal behaviors and the proposed application of LLMs to this dataset comparing results of Zero-Shot and Few-Shot. Our method leverages the context of the collected abnormal activity data to prompt LLMs to suggest window size, overlap rate, and LSTM model’s length sequence tailored to the specific characteristics of these activities. The dataset includes labeled video data collected for four days from five normal participants performing eight activities with four abnormal behaviors. The data was collected with normal participants to simulate activities, and no individuals with disabilities. For evaluation, we assessed all normal versus abnormal activities and per abnormal activity recognition comparing with the baseline without LLM. The results showed that Few-Shot prompting delivered the best performance, with F1-score improvements of 7.69% for throwing things, 7.31% for attacking, 4.68% for head banging, and 1.24% for nail biting as compared to the baseline. Zero-Shot prompting also demonstrated strong recognition capabilities, achieving F1 scores above 96% across all abnormal behaviors. By using LLM-driven suggestions with YOLOv7 pose data, we optimize temporal parameters, enhancing sensitivity to abnormal behaviors and generalization across activities. The model reliably identifies short, complex behaviors, making it ideal for real-world caregiving applications.

View full abstract

Recent advances in ubiquitous sensing technologies have enabled systems to recognize and support daily human activities for behavioral change interventions. While previous research has explored activity recognition in various healthcare domains, automated monitoring and support systems for daily skin care activities remain understudied. This paper presents a novel system that combines continuous sensor monitoring of skin conditions with personalized behavior change support. The system employs precision sensors to capture fine-grained temporal changes in skin moisture and sebum levels, enabling detailed activity recognition of users’ skin care routines and their effects. Based on the collected multimodal sensor data and identified behavioral patterns, the system implements an adaptive intervention mechanism that provides personalized recommendations aligned with individual skin types and care activities. Our research aims to promote sustainable behavior changes in skin care routines by helping users objectively understand the relationship between their daily activities and skin condition variations. Through experimental evaluation, we demonstrate how the system effectively recognizes skin care activities and influences user behavior for improved skin health maintenance. This work contributes to expanding activity recognition and behavior computing applications in the personal care domain.

View full abstract

To alleviate the tension that performers feel during a performance, we proposed a system that judges the audience’s laughter in real-time using machine learning and conveys the feedback to the performers. The conventional threshold-based laughter judgment was insufficient to alleviate tension.Therefore, this study adopted a method to accurately judge laughter using machine learning and reduce stress by providing vibrational feedback. In this experiment, changes in the tension level of a comedic comic performer were evaluated using the system, and a statistically significant tension-relieving effect was obtained. The questionnaire results also suggested areas for improvement, such as the usefulness of visual feedback. Besides, a comparison of the actual laughter timing and the laughter judgment using machine learning showed that the system could recognize laughter at approximately 60% the exact timing.

View full abstract

A 2023 survey of over 4,000 regular fitness enthusiasts revealed that wearable devices have remained a prominent topic. Studies have shown that planned fitness activities and long-term progress tracking can enhance motivation and provide accurate fitness evaluations. However, most current fitness tracking applications require manual data entry before and after workouts, potentially distracting users and reducing workout effectiveness. Few applications can automatically record diverse fitness movements for extended periods. To address these challenges, this study aims to develop a system using Android-based smartwatches and smartphones to recognize and quantify users’ fitness-related movements automatically. By eliminating manual operation, the system offers long-term feedback on fitness activities. The research comprises four key components: (1) Training two machine learning models to recognize motion states and fitness movements with feature dimensionality reduction for real-time mobile operation; (2) Proposing the Double-Layer Sliding Window method to recognize and count fitness movements during exercise based on sliding windows and peak detection; (3) Developing an algorithm for automatic fitness movement recognition and counting in long-term exercise environments; (4) Conducting experiments in Python environments and analyzing statistical data. Results demonstrated the potential of this approach in automatic fitness activity recording.

View full abstract

By analyzing sports movements, performance and skill evaluation can be performed, which can assist players in improving their skills. In basketball shooting, joint movement is essential, particularly on the elbow, shoulder, and wrist. Most previous studies have relied on camera-based systems, which can be costly and environmentally dependent. Wearable device-based systems are also limited in evaluation metrics, often lacking sufficient data for technical support. This study focuses on beginner basketball players, aiming to guide them toward a perfect shooting form by providing feedback on the forearm angle in the set position to achieve an optimal forearm angle. The ideal angle was set at 32±5 degrees based on the forearm angles observed during shots by three experienced players. This system utilizes a smartwatch and smart glass. The smartwatch measures the forearm angle, and based on this data, smart glass provides real-time feedback. When the angle is ideal, the screen shows green; when not, it shows red, prompting the user to adjust the arm angle until it turns green. In this experiment, 20 inexperienced players shot 10 regular and 10 free throws using the system. The evaluation metrics included the forearm angle at the set position, its variability, and the system’s effectiveness as measured by the System Usability Scale (SUS). As a result, when using the system, all participants achieved the ideal forearm angle at the set position, improving their shooting form during the set phase. In forearm angle variability, most players could reduce variability, leading to a more stable shooting form. The SUS evaluation showed an average score of 73.8, indicating good usability. Post-experiment surveys revealed minimal discomfort during shooting due to the devices, suggesting a minor impact from wearing the devices. Many players reported feeling an improvement in shooting form by focusing on the arm angle, confirming the training effectiveness of the system from a subjective perspective. As prospects, Since some subjects found it difficult to hold the position above their heads during the set, it is necessary to examine whether feedback during the hold and during the set would help them acquire the ideal shooting form. Furthermore, creating a system to suggest the appropriate force for throwing the ball could further improve shooting accuracy.

View full abstract

In this paper, we propose a modular framework that integrates fewshot and Generated Knowledge Prompting (FS-GKP) for health information extraction and summarization from nurse-elderly conversation transcripts. This tasks is essential for monitoring elderly patients and assisting nurses in completing the visiting nurse form. FS-GKP generates additional domain-specific knowledge from transcription data, which serves as the basis for more accurate extraction and summarization. FS-GKP uses a structured chain of prompts that allows each step to build on the previous step, thus improving interpretability and precision. Experiments reveal that the GKP using few-shot technique significantly enhances extraction performance with average accuracy across all health categories is 78.57%, outperforming individual methods like zero-shot (52.49%) and few-shot (45.24%). FS-GKP also provides the best results for the summarization task compared to the other five techniques (zero-shot, few-shot, Chain-of-Thouth (CoT), Self-consistency, Few-shot CoT) with ROUGE-1: 0.43, ROUGE-2: 0.22, ROUGE-L: 0.32, BLEU: 0.28, BERTScore Precision: 0.75, Recall: 0.72, F1: 0.73, and SBERT Cosine Similarity: 0.83. These results highlight the potential of FS-GKP, to improve the accuracy of health information extraction and streamline the summarization process, effectively aligning it with categories in visiting nurse forms.

View full abstract