Deep convolutional neural networks (DNNs) have achieved outstanding performance in this field. Meanwhile, handwritten Chinese character recognition (HCCR) is a challenging area of research in the field of computer vision. DNNs require a large number of parameters and high memory consumption. To address these issues, this paper proposes an approach based on an attention mechanism and knowledge distillation. The attention mechanism improves the feature extraction and the knowledge distillation reduces the number of parameters. The experimental results show that ResNet18 achieves a recognition accuracy of 97.63% on the HCCR dataset with 11.25 million parameters. Compared with other methods, this study improves the performance for HCCR.

This paper presents a comparative analysis of classification approaches in the Japanese discourse relation analysis (DRA) task. In the Japanese DRA task, it is difficult to resolve implicit relations where explicit discourse phrases do not appear. To understand implicit relations further, we compared the four approaches by incorporating a special token to encode the relations of the given discourses. Our four approaches included inserting a special token at the beginning of a sentence, end of a sentence, conjunctive position, and random position to classify the relation between the two discourses into one of the following categories: CAUSE/REASON, CONCESSION, CONDITION, PURPOSE, GROUND, CONTRAST, and NONE. Our experimental results revealed that special tokens are available to encode the relations of given discourses more effectively than pooling-based approaches. In particular, the random insertion of a special token outperforms other approaches, including pooling-based approaches, in the most numerous CAUSE/REASON category in implicit relations and categories with few instances. Moreover, we classified the errors in the relation analysis into three categories: confounded phrases, ambiguous relations, and requiring world knowledge for further improvements.

Understanding traditional culture is important. Various methods are used to achieve better cross-cultural understanding, and certain researchers have studied human behavior. However, behavior does not always represent a culture. Therefore, our study aims to understand Japanese greeting culture by classifying it through machine learning. Following are our study contributions. (1) The first study to analyze cultural differences in greeting gestures based on the politeness level of Japanese people by classifying them. (2) Classify Japanese greeting gestures eshaku, keirei, saikeirei, and waving hand. (3) Analyze the performance results of machine and deep learning. Our study noted that bowing and waving were the behaviors that could symbolize the culture in Japan. In conclusion, first, this is the first study to analyze the eshaku, keirei, saikeirei, and waving hand greeting gestures. Second, this study complements several human activity recognition studies that have been conducted but do not focus on behavior representing a culture. Third, according to our analysis, by using a small dataset, SVM and CNN methods provide better results than k-nearest neighbors (k-NN) with Euclidean distance, k-NN with DTW, logistic regression and LightGBM in classifying greeting gestures eshaku, keirei, saikeirei, and waving hand. In the future, we will investigate other behaviors from different perspectives using another method to understand cultural differences.

This research uses a real-time, human-computer interaction application to examine sign language recognition. This work develops a rule-based hand gesture approach for Indonesian sign language in order to interpret some words using a combination of hand movements, mimics, and poses. The main objective in this study is the recognition of sign language that is based on hand movements made in front of the body with one or two hands, movements which may involve switching between the left and right hand or may be combined with mimics and poses. To overcome this problem, a research framework is developed by coordinating hand gestures with poses and mimics to create features by using holistic MediaPipe. To train and test data in real time, the long short time memory (LSTM) and gated recurrent unit (GRU) approaches are used. The research findings presented in this paper show that hand gestures in real-time interactions are reliably recognized, and some words are interpreted with the high accuracy rates of 94% and 96% for the LSTM and GRU methods, respectively.

Robots that learn to perform actions using reinforcement learning to should be able to learn not only static environments, but also environmental changes. Heterogeneous multi-agent reinforcement learning (HMARL) was developed to perform an efficient search, with multiple agents mounted on a single robot to achieve tasks quickly. Responding to environmental changes using normal reinforcement learning can be challenging. However, HMARL does not consider the use of multiple agents to address environmental changes. In this study, we filtered the agents in HMARL using information entropy to realize a robot capable of maintaining high task achievement rates in response to environmental changes.

With the advancement of soft measurement, just-in-time learning (JITL) has become a widely adopted framework for online soft-sensing modeling in industrial processes. However, traditional JITL model approaches often rely on simple similarity measures like Euclidean distance, resulting in the underutilization of labeled data. This paper proposes a supervised, improved local Fisher discriminant analysis method based on a JITL framework and local Fisher discriminant analysis (LFDA) to improve data utilization efficiency. In particular, by incorporating the indirect correlation information matrix, this method integrates the inter-class and intra-class dispersion matrix, overcoming the limitation of the LFDA algorithm that only captures direct data correlations. We select two different carbon depositions in the Methanol-to-Olefin reaction system for comparative experiments and use the root mean squared error (RMSE) and R-square (R²) to evaluate the effectiveness of the proposed method. Fitting results show that two kinds of carbon depositions were better than the control model, namely the RMSE of the model were 0.1431 and 0.1513, R² were 0.8952 and 0.8707.

This paper proposes a heat demand prediction model that analyzes the heat behavior of heat users, and analyzes the heat behavior data of heat users through a clustering algorithm to predict their heat demand. This paper is based on the supply-demand balance strategy to reduce the heat loss during the transmission of heat medium, and then improve the energy-saving efficiency of the boiler. The traditional boiler energy-saving and consumption reduction method is to optimize the boiler combustion parameters, improve the fuel combustion efficiency and waste heat recovery technology through the Internet of Things and big data technology. The method of balancing the heat-using end with the load of the boiler at low usage frequency is seldom considered. Therefore, this paper predicts the heat demand of the heat-using end by analyzing its thermal behavior, and balances the heat demand and boiler heat supply under the condition of meeting the heat demand. Finally, through simulation experiments, the validity of the model is verified, and the trend and data can be well predicted in the short-term heat demand prediction.

This study examines the effect of menu items placed around a slideshow at the center of a webpage on an information search. Specifically, the study analyzes eye movements of users whose search time is long or short on a mixed-type landing page and considers the cause in relation to “directed search” (which triggers a certain type of mental workload). To this end, a Bayesian network model is developed to elucidate the relation between eye movement measures and search time. This model allows the implementation degree of directed search to be gauged from the levels of the measures that characterize a long or short search time. The model incorporates probabilistic dependencies and interactions among eye movement measures, and hence it enables the association of various combinations of these measure levels with different browsing patterns, helping judge whether directed search is implemented or not. When viewers move their eyes in the direction opposite (identical) to the side on which the target information is located, the search time increases (decreases); this movement is a result of the menu items around the slideshow capturing viewers’ attention. However, viewers’ browsing patterns are not related to the initial eye movement directions, which may be classified into either a series of orderly scans (directed search) to reach the target or long-distance eye movements derived from the desire to promptly reach the target (undirected search). These findings suggest that the menu items of a website should not be basically placed around a slideshow, except in cases where they are intentionally placed in only one direction (e.g., left, right, or below).

Herein, the time series storage method of multi-valued attribute data, aimed at improving the efficiency of writing and querying data in “energy” big data centers is reported. Through rule construction and rule iteration of feature sequence, the time series features of multi-valued attribute data are extracted, and the “component attribute nearest neighbor propagation method” is used for clustering; the data are divided into cold, warm, and hot data. A storage engine with a solid state disk layer, mechanical hard disk layer, and memory layer has been designed, and the efficiency of data writing and query through migration operation is improved. The experimental results demonstrate that this method can effectively extract the time series features of multi-valued attribute data, and the Pre value of the clustering time series is higher than 0.93, which effectively improves the data writing and query efficiency of the energy big data center.

In the new, dry-process method of cement production, the temperature of cement rotary kiln sintering zone is a key factor in ensuring the quality of cement clinker. Based on the auto-regressive with extra inputs model, a finite control set model predictive control with soft constraint of the generalized triangular interval is proposed in this paper for the characteristics of a cement rotary kiln calcination system with multi-variable, multi-time delay, bounded disturbance, and slow dynamic process. Simulation experiments show that the steady-state error of the control algorithm proposed in this paper is smaller with better anti-disturbance performance than that of the traditional reference-trajectory-constrained, predictive control algorithm.

Methods for interval priority weight estimation from a crisp pairwise comparison matrix were proposed in the interval analytic hierarchy process assuming the vagueness of human evaluation. The interval priority weights estimated by the conventional method do not reflect the intrinsic vagueness in the given pairwise comparison matrix (PCM). This paper proposes parameter-free methods based on minimal conceivable ranges for estimating interval priority weights from a crisp pairwise comparison matrix. The estimated interval priority weight vectors are required to satisfy (1) the potential reproducibility, (2) the normality, and (3) the preservation of the perfect consistent data. Estimation methods of interval priority weights are proposed based on the minimum possible range. We show those proposed methods satisfy the required three properties. The estimation problem of interval priority weights potentially has multiple solutions with which the associated interval PCMs are identical to one another. To make the further investigation simpler, we use an interval priority weight vector among multiple solutions such that the sum of the center values of interval priority weights is one. We compare the estimation methods of interval priority weights from the viewpoint of estimation accuracy by numerical experiments. Namely, by generating crisp pairwise comparison matrices randomly under true interval PCMs, we evaluate the accuracies of the estimated interval priority weight vectors by comparing the true interval priority weight vectors.

The estimation of leaf area is an important measure for understanding the growth, development, and productivity of tomato plants. In this study, we focused on the leaf area of a potted tomato plant and proposed methods, namely, NP, D², and D³, for estimating its leaf area. In the NP method, we used multiple tomato plant images from different viewing angles to reduce the estimation error of the leaf area, whereas in the D² and D³ methods, we further compensated for the perspective effects. The performances of the proposed methods were experimentally assessed using 40 “Momotaro Peace” tomato plants. The experimental results confirmed that the NP method had a smaller mean absolute percentage error (MAPE) on the test set than the conventional estimation method that uses a single tomato plant image. Likewise, the D² and D³ methods had a smaller MAPE on the test set than the conventional method that did not compensate for perspective effects.

Relation extraction is a fundamental task in natural language processing that aims to identify structured triple relationships from unstructured text. In recent years, research on relation extraction has gradually advanced from the sentence level to the document level. Most existing document-level relation extraction (DocRE) models are fully supervised and their performance is limited by the dataset quality. However, existing DocRE datasets suffer from annotation omission, making fully supervised models unsuitable for real-world scenarios. To address this issue, we propose the DocRE method based on uncertainty pseudo-label selection. This method first trains a teacher model to annotate pseudo-labels for a dataset with incomplete annotations, trains a student model on the dataset with annotated pseudo-labels, and uses the trained student model to predict relations on the test set. To mitigate the confirmation bias problem in pseudo-label methods, we performed adversarial training on the teacher model and calculated the uncertainty of the model output to supervise the generation of pseudo-labels. In addition, to address the hard-easy sample imbalance problem, we propose an adaptive hard-sample focal loss. This loss can guide the model to reduce attention to easy-to-classify samples and outliers and to pay more attention to hard-to-classify samples. We conducted experiments on two public datasets, and the results proved the effectiveness of our method.

The increasing elderly population presents significant challenges in terms of the meeting of their daily care needs. Cognitive decline and reduced arm reflexes following balance loss impede the elderly’s execution of activities of daily living. To address these challenges, robots have emerged as valuable assistants for elderly individuals in their daily activities, including object manipulation, and have the potential to significantly improve the quality of life for the aging population. However, no research has been undertaken to enhance the selection of object handover locations in human-robot interactions by merging topology mapping with both parties’ range of motion, based on personal space. Based on the idea of personal space within human-robot proxemics, this research presents an alternative approach that makes use of topological mapping while taking into account the range of motion of both humans and robots. This research aims to minimize the expenses related to human-robot proximity and to determine the best locations for item handovers in order to discover which locations are optimal. In order to improve object handover locations, this work is a groundbreaking attempt to combine growing neural gas and human proxemics inside a robotic framework. Furthermore, it implies the creation of robot behaviors that resemble human proximity by estimating personal distances and incorporating rule-based requirements for item handover locations by taking into account the mobility ranges of both humans and robots. The simulation findings reported in this work show the ability of the suggested methodology and offer interesting information and prospects for further developments in the area of object handovers by robots.

Since deep Q-networks and AlphaGO by Google DeepMind were proposed, not only a reinforcement learning integrated with deep learning but also its applications have attracted much attention on. ChatGPT, which learns by Proximal Policy Optimization as one of reinforcement learning mechanisms, is an excellent example.

With this background, we propose the special issue titled “Cutting Edge of Reinforcement Learning and its Hybrid Methods,” which is the same of the special issue in 2017, to cover the latest progress and trends in reinforcement learning and its hybrid methods (combined with machine learning, neural networks, and evolutionary computation).

We received 28 submissions, out of which 12 submissions were peer-reviewed and 16 submissions that were outside the scope of the special issue were excluded. After a thorough review process including comments and suggestions by reviewers, seven submissions were accepted. These included four theory and three application papers, indicating nearly equal contributions in theory and application aspects.

The content of the first three papers is related to inverse reinforcement learning, while the fourth one is not directly related to reinforcement learning but it would be related to reinforcement learning in the future. All papers address issues surrounding cutting-edge technology in reinforcement learning.

We would like to end by expressing that we hope and believe that this special issue can largely contribute to the development in the field of reinforcement learning while holding a wide international appeal.

Inverse reinforcement learning (IRL) estimates a reward function for an agent to behave along with expert data, e.g., as human operation data. However, expert data usually have redundant parts, which decrease the agent’s performance. This study extends the IRL to sub-optimal action data, including lack and detour. The proposed method searches for new actions to determine optimal expert action data. This study adopted maze problems with sub-optimal expert action data to investigate the performance of the proposed method. The experimental results show that the proposed method finds optimal expert data better than the conventional method, and the proposed search mechanisms perform better than random search.

Sequential decision-making under multiple objective functions includes the problem of exhaustively searching for a Pareto-optimal policy and the problem of selecting a policy from the resulting set of Pareto-optimal policies based on the decision maker’s preferences. This paper focuses on the latter problem. In order to select a policy that reflects the decision maker’s preferences, it is necessary to order these policies, which is problematic because the decision-maker’s preferences are generally tacit knowledge. Furthermore, it is difficult to order them quantitatively. For this reason, conventional methods have mainly been used to elicit preferences through dialogue with decision-makers and through one-to-one comparisons. In contrast, this paper proposes a method based on inverse reinforcement learning to estimate the weight of each objective from the decision-making sequence. The estimated weights can be used to quantitatively evaluate the Pareto-optimal policies from the viewpoints of the decision-makers preferences. We applied the proposed method to the multi-objective reinforcement learning benchmark problem and verified its effectiveness as an elicitation method of weights for each objective function.

In recent years, inverse reinforcement learning has attracted attention as a method for estimating the intention of actions using the trajectories of various action-taking agents, including human flow data. In the context of reinforcement learning, “intention” refers to a reward function. Conventional inverse reinforcement learning assumes that all trajectories are generated from policies learned under a single reward function. However, it is natural to assume that people in a human flow act according to multiple policies. In this study, we introduce an expectation-maximization algorithm to inverse reinforcement learning, and propose a method to estimate different reward functions from the trajectories of human flow. The effectiveness of the proposed method was evaluated through a computer experiment based on human flow data collected from subjects around airport gates.

In lowest unique integer games (LUIGs), continually choosing the same number has been experimentally and computationally shown to be effective. However, this result holds only when all players behave differently, and it is unclear whether such behavior performs well under population dynamics. This study analyzed the types of agents that survive and how successfully they behave within an evolutionary environment of small-size LUIGs. Here, the author identified a learning model to create agents using behavioral data obtained from a laboratory experiment by Yamada and Hanaki. Then, evolutionary competition was pursued. The main findings are three fold. First, more agents are ruled out in three-person LUIGs than in four-person LUIGs. Second, the most successful agents do not win as much as the generations increase. Instead, they manage to win by adaptively changing their strategies. Third, as the scale of the LUIG increases, the number of wins for each agent is not correlated with that in a round-robin contest.

In recent years, studies on practical application of automated driving have been conducted extensively. Most of the research assumes the existing road infrastructure and aims to replace human driving. There have also been studies that use reinforcement learning to optimize car control from a zero-based perspective in an environment without lanes, one of the existing types of road. In those studies, search and behavior acquisition using reinforcement learning has resulted in efficient driving control in an unknown environment. However, the throughput has not been high, while the crash rate has. To address this issue, this study proposes a hierarchical reward model that uses both individual and common rewards for reinforcement learning in order to achieve efficient driving control in a road, we assume environments of one-way, lane-less, automobile-only. Automated driving control is trained using a hierarchical reward model and evaluated through physical simulations. The results show that a reduction in crash rate and an improvement in throughput is attained by increasing the number of behaviors in which faster cars actively overtake slower ones.

To increase an accuracy of the sleep stage estimation without connecting any devices/electrodes to the body, this paper proposes the updating method for its estimation according to an ultradian rhythm as one of the biological rhythms of humans. In the proposed method, the prediction probability of the sleep stage is updated by the Widrow–Hoff learning rule which is generally employed in the update of reinforcement learning. Through the human subject experiment which acquired the biological vibration data from the mattress sensor during sleep, the following implications have been revealed: (1) the accuracy and the quadratic weighted kappa (QWK) of the sleep stage estimation updated by the proposed method are higher than those of random forest (RF) as the conventional method; (2) the multiple update of the probability of the sleep stage according to the ultradian rhythm is significantly important to improve its accuracy and QWK; and (3) compared with RF which over-estimated the NR2 stage while less-estimated the NR1 and NR3 stages, the proposed method contributes to correctly estimating the NR1–3 stages thank to the follow of the ultradian rhythm.

The National Institution for Academic Degrees and Quality Enhancement of Higher Education (NIAD-QE) awards academic degrees based on credit accumulation. These credits must be classified according to predetermined criteria for the selected disciplinary fields. This study was conducted by subcommittees within the Committee for Validation and Examination of Degrees, the members of which should be well-versed in the syllabus of each course to ensure appropriate classification. The number of applicants has been increasing annually, and thus, a course-classification system supported by information technology is strongly desired. We proposed a course-classification support system (CCS) and an active CCS system for awarding degrees in NIAD-QE. In contrast, in this study, we construct a CCS using deep learning, which has been significantly developed in recent years. We also propose a method “CLCNNwithXoL” combined with the reinforcement learning method. We evaluate its effectiveness using the data submitted.