Verification of general-purpose language models and deep learning models for estimating brain activities evoked by verbal stimuli

Abstract

For the estimation of brain states in spoken conversation stimuli, we conducted an experiment using three types of deep learning models (Bi-LSTM/Bi-GRU/Bi-RNN) to estimate brain activity data using speech spectrogram as speech features, and compared the estimation performance of each model. There was no significant difference in the performance of any of those models, and we confirmed that the brain regions close to the ears, which are considered to be responsible for phonological and grammatical processing, responded better. In addition, we predicted brain activity using linguistic features transcribed from auditory stimuli into text. We used RoBERTa/BERT/word2vec as a general-purpose language model to convert them into embedded vectors. In this experiment, we could confirm responses in a wide range of language areas in the brain, not limited to the peripheral regions of the ear.