An Approach to Extracting Linguistic Information in the Human Brain Activity evoked by Speech Stimuli

Abstract

The semantic activities in the human brain has been actively studied in the field of neuroscience. In this paper, we propose a deep learning method to describe the semantic activities in the brain evoked by speech stimuli observed by from Functional Magnetic Resonance Imaging (fMRI). Thereby, our study aims to decode higher order perception, i.e., semantic representation, which a person recalled in the brain when speech stimuli were given to him or her. However, collecting a large-scale brain activity dataset is difficult because observing brain activity data with fMRI is expensive, although a method with deep learning requires a large-scale dataset. We therefore use an automatic speech recognition method and utilize a small amounts of fMRI data efficiently for building a model. Through experiments, we have confirmed that high correlation exists between the predicted features from fMRI data and the speech features.