Analysis of Correspondence Relationship between Brain Activity and Semantic Representation

Abstract

It is known that primary visual cortex uses a sparse code to efficiently represent natural scenes. Based on this fact, we built up a hypothesis that the same phenomenon happens at the higher cognitive function. Here we focus on semantic representation reflecting the meaning of words in the cerebral cortex. We applied sparse coding to the matrix consisting of paired data for both brain activity evoked by visual stimuli observed while a subject is watching a video, and distributed semantic representation made from the description of the video by means of a word2vec language model. Using this method, we obtained a dictionary matrix whose bases represent the corresponding relation between brain activity and the semantic representation. We then analyzed the characteristics of each base in the dictionary matrix. As a result, we confirmed that independent perceptual units were extracted with words representing their functional meaning.