fMRIを用いた深層学習による脳聴覚野からの周波数スペクトルに着目した楽器の音色の推定

抄録

The purpose of this study is to establish a technique for estimating sounds heard by humans from brain images captured by fMRI using deep learning. In our previous report, we found that CNN, a type of deep learning, can estimate the pitch of single tones in pure tones and specific tones in chords. On the other hand, sounds we hear in our daily lives are compound waves containing multiple frequencies, and their timbre is determined by their frequency spectrum. Therefore, we suppose it would be possible to estimate the timbre using the same method as in the previous report. In this report, we focused on the relationship between the sound pressure levels of each harmonic in the frequency spectrum and used 3DCNN to estimate the timbre of the musical instrument heard by each volunteer from brain activation images obtained when hearing to two different musical instruments. As a result, the accuracy of the estimation was not sufficient.