Towards Understanding the Hierarchical Processing of Visual and Linguistic Information in the Human Brain

Abstract

This study aims to elucidate the mechanism for hierarchical information processing in the human brain by modeling brain information using two types of deep learning models: a convolutional neural network (CNN), which is a hierarchical model for image processing, and BERT, which is a general-purpose language model with Transformers. We constructed encoding models for predicting brain activity from feature representations in each layer of the deep learning models and applied representational similarity analysis (RSA) and PageRank algorithm to the predicted brain activity to examine the change in the network hub structure of brain regions during hierarchical information processing. As for the visual processing, we found that the hubness of the occipital visual cortex increases in the early visual processing whereas the hubness of the prefrontal and temporal cortices increases in the late visual processing.