A Study of the Localization and Representation of Visual and Language Information in the Human Brain

Abstract

In recent years, many studies have been conducted to elucidate the information processing mechanism of the human brain using deep learning. In this study, to investigate the relationship between visual and language information in the human brain, we conducted encoding modeling to predict brain activity based on features of videos extracted from the hidden layers of deep learning models, convolutional neural networks for visual information, Transfomer for language information, and a multimodal processing model, CLIP. Then, we analyze brain states predicted by the encoding models to investigate the hierarchical properties of the localization and representational content of visual and language information in the cerebral cortex. As a result, we found that the similarities in predictable brain regions are similar to those in spatial information representation content, but not similar to those in temporal information representation content.