Article ID: 2025LHP0002
This study proposes a novel analog integrated circuit to solve the two-armed bandit (2AB) problem, which is a fundamental decision-making task in reinforcement learning. The 2AB problem involves repeatedly selecting the better of two uncertain options to maximize cumulative rewards. While conventional implementations have relied on software or digital circuits, limitations in clock frequency and memory access latency have constrained further improvements in processing speed. To address this issue, we propose a circuit architecture that leverages the physical characteristics of analog components for high-speed operation. The proposed circuit is fabricated using TSMC 180-nm CMOS process and integrates a dynamic comparator along with a charge pump-based history accumulation circuit (HAC). This configuration enables real-time adaptation to changes in reward probability based on the Thompson sampling strategy. In addition, a pulse generation circuit is introduced to clearly separate the phases of selection and learning, thereby suppressing timing mismatches and hazards to ensure stable operation. The prototype chip was evaluated under various reward probability settings and with a clock frequency of up to 1.0 GHz. Experimental results confirmed correct learning behavior at frequencies up to 840 MHz. However, a certain degree of selection imbalance caused by asymmetry in the charge and discharge operations of the HAC was observed. Future work will focus on improving the charge/discharge balance of the HAC and optimizing the comparator design to further enhance performance.
