Analyzing Neurons Engaged in Multi-step Reasoning in Pre-trained Language Models

Abstract

Prompts have attracted attention as a way to exploit the performance of pre-trained language models, one of which is the chain-of-thought prompt. Chain-of-thought prompts are prompts that encourage the explicit expression of intermediate thoughts in order to derive a final answer, and have attracted attention for their ability to improve multi-stage reasoning. On the other hand, it remains unclear how chain-of-thought prompts affect models and enable multi-step reasoning. In this paper, we examine how neurons in models are internally influenced in multi-step reasoning tasks, against the background of existing studies that interpret task performance based on the activation of neurons in language models. The results revealed that there are neurons that are commonly activated in multiple chain-of-thought prompts in multi-step reasoning. We also found that suppressing the activation of these neurons worsened reasoning performance. These results have implications for the mechanisms by which models acquire reasoning ability.