Decoding of semantic representations in the brain under language stimulation using language models

Abstract

In many brain decoding studies, high-resolution invasive data is often preferred for superior performance. However, Tang et al. introduced a non-invasive approach for reconstructing sentences by generating output candidates from a language model and selecting the one that best aligns with cortical semantic representations obtained via fMRI. This study aims to further validate the decoding method proposed by Tang et al. by employing a more advanced language model (Llama3). The objective is to confirm the effectiveness of the conventional method and to elucidate the relationship between brain representation patterns and text representations in greater detail.