The empirical characteristics of domestic and foreign interest rate shocks are obtained by using VAR method: the domestic interest rate regulation is counter-cyclical, and the increase of foreign interest rate leads to the increase of domestic output and inflation. On this basis, we construct a small open dynamic stochastic general equilibrium theory framework which reflects the empirical characteristics, including exchange rate control, to analyze the macroeconomic effects of exchange rate liberalization reform. By volatility simulation, impulse response and social welfare loss function analysis, the empirical results show that: firstly, exchange rate reform would increase volatility of output and exchange rate, but reduce volatility of inflation and interest rate. Secondly, exchange rate reform enhances the impact of domestic interest rate shocks on output and inflation. Which means the reform would improve the control ability of interest rate as a monetary policy tool. Moreover, the reform increases loss of social welfare. The conclusion shows that the exchange rate liberalization should be implemented step by step. The government should accelerate the reform when the external macro economy is stable. Otherwise it will cause a larger economic volatility.

This study analyses the robustness and convergence characteristics of a neural network. First, a special class of recurrent neural network (RNN), termed a continuous-time Zhang neural network (CTZNN) model, is presented and investigated for dynamic matrix pseudoinversion. Theoretical analysis of the CTZNN model demonstrates that it has good robustness against various types of noise. In addition, considering the requirements of digital implementation and online computation, the optimal sampling gap for a discrete-time Zhang neural network (DTZNN) model under noisy environments is proposed. Finally, experimental results are presented, which further substantiate the theoretical analyses and demonstrate the effectiveness of the proposed ZNN models for computing a dynamic matrix pseudoinverse under noisy environments.

Sintering is an important production process in iron and steel metallurgy. Carbon fuel consumption accounts for about 80% of the total energy consumption in the sintering process. To enhance the efficiency of carbon fuel consumption, we need to determine the factors affecting carbon efficiency and build a model of it. In this paper, the CO/CO₂ is taken to be a measure of carbon efficiency, and a cascade predictive model is built to predict it. This model has two parts: the key state parameter submodel and the CO/CO₂ submodel. The submodels are built using particle swarm optimization-based back propagation neural networks (PSO-BPNNs). Based on the mechanism analysis, spearman’s rank correlation coefficient (SRCC) and stepwise regression analysis (SRA) are used to determine the relationship between the process parameters, in order to determine the inputs of each submodel. Finally, the results of a simulation show the feasibility of the cascade model, which will serve as the basic model for the optimization and control of the carbon efficiency of the sintering process.

Conventional kernel support vector machine (KSVM) has the problem of slow training speed, and single kernel extreme learning machine (KELM) also has some performance limitations, for which this paper proposes a new combined KELM model that build by the polynomial kernel and reproducing kernel on Sobolev Hilbert space. This model combines the advantages of global and local kernel function and has fast training speed. At the same time, an efficient optimization algorithm called cuckoo search algorithm is adopted to avoid blindness and inaccuracy in parameter selection. Experiments were performed on bi-spiral benchmark dataset, Banana dataset, as well as a number of classification and regression datasets from the UCI benchmark repository illustrate the feasibility of the proposed model. It achieves the better robustness and generalization performance when compared to other conventional KELM and KSVM, which demonstrates its effectiveness and usefulness.

The major objective of the paper is to investigate a new probabilistic supervised learning approach that incorporates “missingness” into a decision tree classifier splitting criterion at each particular attribute node in terms of software effort development predictive accuracy. The proposed approach is compared empirically with ten supervised learning methods (classifiers) that have mechanisms for dealing with missing values. 10 industrial datasets are utilized for this task. Overall, missing incorporated in attributes 3 is the top performing strategy, followed by C4.5, missing incorporated in attributes, missing incorporated in attributes 2, missing incorporated in attributes, linear discriminant analysis and so on. Classification and regression trees and C4.5 performed well in data with high correlations among attributes while k-nearest neighbour and support vector machines performed well in data with higher complexity (limited number of instances). The worst performing method is repeated incremental pruning to produce error reduction.

In recent years, many studies have been performed on the automatic classification of human body motions based on inertia sensor data using a combination of inertia sensors and machine learning; training data is necessary where sensor data and human body motions correspond to one another. It can be difficult to conduct experiments involving a large number of subjects over an extended time period, because of concern for the fatigue or injury of subjects. Many studies, therefore, allow a small number of subjects to perform repeated body motions subject to classification, to acquire data on which to build training data. Any classifiers constructed using such training data will have some problems associated with generalization errors caused by individual and trial differences. In order to suppress such generalization errors, feature spaces must be obtained that are less likely to generate generalization errors due to individual and trial differences. To obtain such feature spaces, we require indices to evaluate the likelihood of the feature spaces generating generalization errors due to individual and trial errors. This paper, therefore, aims to devise such evaluation indices from the perspectives. The evaluation indices we propose in this paper can be obtained by first constructing acquired data probability distributions that represent individual and trial differences, and then using such probability distributions to calculate any risks of generating generalization errors. We have verified the effectiveness of the proposed evaluation method by applying it to sensor data for butterfly and breaststroke swimming. For the purpose of comparison, we have also applied a few available existing evaluation methods. We have constructed classifiers for butterfly and breaststroke swimming by applying a support vector machine to the feature spaces obtained by the proposed and existing methods. Based on the accuracy verification we conducted with test data, we found that the proposed method produced significantly higher F-measure than the existing methods. This proves that the use of the proposed evaluation indices enables us to obtain a feature space that is less likely to generate generalization errors due to individual and trial differences.

Gaussian process dynamical models (GPDMs) are used for nonlinear dimensionality reduction in time series by means of Gaussian process priors. An extension of GPDMs is proposed for visualizing the states of time series. The conventional GPDM approach associates a state with an observation value. Therefore, observations changing over time cannot be represented by a single state. Consequently, the resulting visualization of state transition is difficult to understand, as states change when the observation values change. To overcome this issue, autoregressive GPDMs, called ARGPDMs, are proposed. They associate a state with a vector autoregressive (VAR) model. Therefore, observations changing over time can be represented by a single state. The resulting visualization is easier to understand, as states change only when the VAR model changes. We demonstrate experimentally that the ARGPDM approach provides better visualization compared with conventional GPDMs.

Machine learning has been attracting significant attention again since the potential of deep learning was recognized. Not only has machine learning been improved, but it has also been integrated with “reinforcement learning,” revealing other potential applications, e.g., deep Q-networks (DQN) and AlphaGO proposed by Google DeepMind. It is against this background that this special issue, “Cutting Edge of Reinforcement Learning and its Hybrid Methods,” focuses on both reinforcement learning and its hybrid methods, including reinforcement learning with deep learning or evolutionary computation, to explore new potentials of reinforcement learning.

Of the many contributions received, we finally selected 13 works for publication. The first three propose hybrids of deep learning and reinforcement learning for single agent environments, which include the latest research results in the areas of convolutional neural networks and DQN. The fourth through seventh works are related to the Learning Classifier System, which integrates evolutionary computation and reinforcement learning to develop the rule discovery mechanism. The eighth and ninth works address problems related to goal design or the reward, an issue that is particularly important to the application of reinforcement learning. The last four contributions deal with multiagent environments.

These works cover a wide range of studies, from the expansion of techniques incorporating simultaneous learning to applications in multiagent environments. All works are on the cutting edge of reinforcement learning and its hybrid methods. We hope that this special issue constitutes a large contribution to the development of the reinforcement learning field.

This paper proposes a new multi-feature detection method of local pedestrian based on a convolutional neural network (CNN), which provides a reliable basis for multi-feature fusion in pedestrian detection. According to the standard of pedestrian detection ratio, the pedestrian under the detection window would be segmented, using the sample labels to guide the local characteristics of CNN learning, the supervised learning after the network can obtain the local feature fusion more pedestrian description ability. Finally, a large number of experiments have been performed. The experimental results show that the local features of the neural network are better than those of most pedestrian features and combination features.

Deep Q-network (DQN) is one of the most famous methods of deep reinforcement learning. DQN approximates the action-value function using Convolutional Neural Network (CNN) and updates it using Q-learning. In this study, we applied DQN to robot behavior learning in a simulation environment. We constructed the simulation environment for a two-wheeled mobile robot using the robot simulation software, Webots. The mobile robot acquired good behavior such as avoiding walls and moving along a center line by learning from high-dimensional visual information supplied as input data. We propose a method that reuses the best target network so far when the learning performance suddenly falls. Moreover, we incorporate Profit Sharing method into DQN in order to accelerate learning. Through the simulation experiment, we confirmed that our method is effective.

Currently, deep learning is attracting significant interest. Combining deep Q-networks (DQNs) and Q-learning has produced excellent results for several Atari 2600 games. In this paper, we propose an exploitation-oriented learning (XoL) method that incorporates deep learning to reduce the number of trial-and-error searches. We focus on a profit sharing (PS) method that is an XoL method, and combine it with a DQN to propose a DQNwithPS method. This method is compared with a DQN in Atari 2600 games. We demonstrate that the proposed DQNwithPS method can learn stably with fewer trial-and-error searches than required by only a DQN.

The correctness rate of classification of neural networks is improved by deep learning, which is machine learning of neural networks, and its accuracy is higher than the human brain in some fields. This paper proposes the hybrid system of the neural network and the Learning Classifier System (LCS). LCS is evolutionary rule-based machine learning using reinforcement learning. To increase the correctness rate of classification, we combine the neural network and the LCS. This paper conducted benchmark experiments to verify the proposed system. The experiment revealed that: 1) the correctness rate of classification of the proposed system is higher than the conventional LCS (XCSR) and normal neural network; and 2) the covering mechanism of XCSR raises the correctness rate of proposed system.

In this paper, we propose a method to improve ECS-DMR which enables appropriate output for imbalanced data sets. In order to control generalization of LCS in imbalanced data set, we propose a method of applying imbalance ratio of data set to a sigmoid function, and then, appropriately update the matching range. In comparison with our previous work (ECS-DMR), the proposed method can control the generalization of the appropriate matching range automatically to extract the exemplars that cover the given problem space, wchich consists of imbalanced data set. From the experimental results, it is suggested that the proposed method provides stable performance to imbalanced data set. The effect of the proposed method using the sigmoid function considering the data balance is shown.

The XCS classifier system is an evolutionary rule-based learning technique powered by a Q-learning like learning mechanism. It employs a global deletion scheme to delete rules from all rules covering all state-action pairs. However, the optimality of this scheme remains unclear owing to the lack of intensive analysis. We here introduce two deletion schemes: 1) local deletion, which can be applied to a subset of rules covering each state (a match set), and 2) stronger local deletion, which can be applied to a more specific subset covering each state-action pair (an action set). The aim of this paper is to reveal how the above three deletion schemes affect the performance of XCS. Our analysis shows that the local deletion schemes promote the elimination of inaccurate rules compared with the global deletion scheme. However, the stronger local deletion scheme occasionally deletes a good rule. We further show that the two local deletion schemes greatly improve the performance of XCS on a set of noisy maze problems. Although the localization strength of the proposed deletion schemes may require consideration, they can be adequate for XCS rather than the original global deletion scheme.

This paper presents an approach to clustering that extends the variance-based Learning Classifier System (XCS-VR). In real world problems, the ability to combine similar rules is crucial in the knowledge discovery and data mining field. Conventionally, XCS-VR is able to acquire generalized rules, but it cannot further acquire more generalized rules from these rules. The proposed approach (called XCS-VRc) accomplishes this by integrating similar generalized rules. To validate the proposed approach, we designed a bench-mark problem to examine whether XCS-VRc can cluster both the generalized and more generalized features in the input data. The proposed XCS-VRc proved to be more efficient than XCS and the conventional XCS-VR.

This paper focuses on the generalization of classifiers in noisy problems and aims at construction learning classifier system (LCS) that can acquire the optimal classifier subset by dynamically determining the classifier generalization criteria. In this paper, an accuracy-based LCS (XCS) that uses the mean of the reward (XCS-MR) is introduced, which can correctly identify classifiers as either accurate or inaccurate for noisy problems, and investigates its effectiveness when used for several noisy problems. Applying XCS and an XCS based on the variance of reward (XCS-VR) as the conventional LCSs, along with XCS-MR, to noisy 11-multiplexer problems where the reward value changes according to a Gaussian distribution, Cauchy distribution, and lognormal distribution revealed the following: (1) XCS-VR and XCS-MR could select the correct action for every type of reward distribution; (2) XCS-MR could appropriately generalize the classifiers with the smallest amount of data; and (3) XCS-MR could acquire the optimal classifier subset in every trial for every type of reward distribution.

This paper proposes a learning goal space that visualizes the distribution of the obtained solutions to support the exploration of the learning goals for a learner. Subsequently, we examine the method for assisting a learner to present the novelty of the obtained solution. We conduct a learning experiment using a continuous learning task to identify various solutions. To assign the subjects space to explore the learning goals, several parameters related to the success of the task are not instructed to the subjects. In the comparative experiment, three types of learning feedbacks provided to the subjects are compared. These are presenting the learning goal space with obtained solutions mapped on it, directly presenting the novelty of the obtained solutions mapped on it, and presenting some value that is slightly related to the obtained solution. In the experiments, the subjects to whom the learning goal space or novelty of the obtained solution is shown, continue to identify solutions according to their learning goals until the final stage in the experiment is attained. Therefore, in a continuous learning task, our supporting method of directly or indirectly presenting the novelty of the obtained solution through the learning goal space is effective.

This study discusses important factors for zero communication, multi-agent cooperation by comparing different modified reinforcement learning methods. The two learning methods used for comparison were assigned different goal selections for multi-agent cooperation tasks. The first method is called Profit Minimizing Reinforcement Learning (PMRL); it forces agents to learn how to reach the farthest goal, and then the agent closest to the goal is directed to the goal. The second method is called Yielding Action Reinforcement Learning (YARL); it forces agents to learn through a Q-learning process, and if the agents have a conflict, the agent that is closest to the goal learns to reach the next closest goal. To compare the two methods, we designed experiments by adjusting the following maze factors: (1) the location of the start point and goal; (2) the number of agents; and (3) the size of maze. The intensive simulations performed on the maze problem for the agent cooperation task revealed that the two methods successfully enabled the agents to exhibit cooperative behavior, even if the size of the maze and the number of agents change. The PMRL mechanism always enables the agents to learn cooperative behavior, whereas the YARL mechanism makes the agents learn cooperative behavior over a small number of learning iterations. In zero communication, multi-agent cooperation, it is important that only agents that have a conflict cooperate with each other.

When multiple agents learn a task simultaneously in an environment, the learning results often become unstable. This problem is known as the concurrent learning problem and to date, several methods have been proposed to resolve it. In this paper, we propose a new method that incorporates expected failure probability (EFP) into the action selection strategy to give agents a kind of mutual adaptability. The effectiveness of the proposed method is confirmed using Keepaway task.

In this paper, we propose a method to adapt the exploration ratio in multi-agent reinforcement learning. The adaptation of exploration ratio is important in multi-agent learning, as this is one of key parameters that affect the learning performance. In our observation, the adaptation method can adjust the exploration ratio suitably (but not optimally) according to the characteristics of environments. We investigated the evolutionarily adaptation of the exploration ratio in multi-agent learning. We conducted several experiments to adapt the exploration ratio in a simple evolutionary way, namely, mimicking advantageous exploration ratio (MAER), and confirmed that MAER always acquires relatively lower exploration ratio than the optimal value for the change ratio of the environments. In this paper, we propose a second evolutionary adaptation method, namely, win or update exploration ratio (WoUE). The results of the experiments showed that WoUE can acquire a more suitable exploration ratio than MAER, and the obtained ratio was near-optimal.

In this paper, we introduce an autonomous decentralized method for directing multiple automated guided vehicles (AGVs) in response to uncertain delivery requests. The transportation route plans of AGVs are expected to minimize the transportation time while preventing collisions between the AGVs in the system. In this method, each AGV as an agent computes its transportation route by referring to the static path information. If potential collisions are detected, one of the two agents chosen by a negotiation-rule modifies its route plan. Here, we propose a reinforcement learning approach for improving the negotiation-rules. Then, we confirm the effectiveness of the proposed approach based on the results of computational experiments.