An Extention of Profit Sharing to Partially Observable Markov Decision Processes : Proposition of PS-r* and its Evaluation

Abstract

We know the rationality theorem of Profit Sharing(PS) [Miyazaki 94, Miyazaki 99b] and the Rational Policy Making algorithm(RPM) [Miyazaki 99a] to guarantee the rationality in a typical class of Partially Observable Markov Decision Processes (POMDPs). In this paper, we focus on the whole class of POMDPs and propose PS-r that is an algorithm connected PS and RPM with random selection. In the first, we have analyzed the behavior of PS-r. We have derived that the maximum value of the step to get a reward by PS-r divided by that of random selection is

$({\Large r\frac{(1+\frac{M-1}{r})^n}{M^n}})$ where $n$

is the maximum number of state that senses same state due to the agent's sensory limitation and M is the number of actions. Furthermore, we propose PS-r* that can improve the behavior of PS-r. Through numerical examples, we conform the effectiveness of PS-r*.