Reconstruction of stimulus sentences using deep sentence generation model

Abstract

The neural basis of our language comprehension system has been explored using neuroimaging techniques, such as functional magnetic resonance imaging. Despite having identified brain regions and systems related to various linguistic information aspects, the entire image of a neurocomputational model of language comprehension remains unsolved. Contrastingly, in machine learning, the rapid development of natural language models using deep learning allowed sentence generation models to generate high-accuracy sentences. Mainly, this study aimed to build a method that reconstructs stimulus sentences directly only from neural representations to evaluate a neurocomputational model for understanding linguistic information using these text generation models. Consequently, the variational autoencoder model combined with pre-trained deep neural network models showed the highest decoding accuracy, and we succeeded in reconstructing stimulus sentences directly only from neural representations using this model. Although we only achieved topic-level sentence generation, we still exploratorily analyzed the characteristics of neural representations in language comprehension, considering this model as a neurocomputational model.