Correspondence between human brain activity and the latent representations of Large Language Models during the semantic comprehension of speech, objects, and stories

Abstract

One of the major goals in Artificial Intelligence research is to construct machine learning models that comprehend semantics as humans do. While Large Language Models (LLMs) have significantly improved the benchmarks in semantic comprehension, how LLMs’ internal representations encode semantic information and their resemblance to the human brain remain poorly understood. This study aims to elucidate these mechanisms by examining the correspondence between human brain activity during semantic comprehension and the latent representations of LLMs. We collected human brain activity using functional magnetic resonance imaging (fMRI) when human subjects watched drama series. We also collected annotations at various levels related to the drama, such as speech, objects, and stories, and we extracted the corresponding latent representations from LLMs. We demonstrate that, especially for higher-level semantic contents, the latent representations of LLMs explain human brain activity more accurately than traditional language models. Additionally, we show that distinct brain regions correspond to different latent representations in LLMs, inferred from the different levels of semantic contents.