In reinforcement learning problems, the learning of action-value functions is carried out in an incremental way. To realize a stable learning in such situations, we have proposed an RBF network model with memory mechanism. This model learns based on a gradient descent method. The gradient learning method is generally slow. To improve this problem, we apply the linear method to the learning of RBF networks. In this algorithm, the centers of basis functions are not trained, but only connection weights are trained. In our computer simulations, it is verified that the proposed model can learn fast and accurately.