Multimodal Concept Learning using Biosignals

Abstract

The brain receives various signals through its own body. These signals are classified into exteroception, interoception, and proprioception and are structurally integrated. This integrated structure is considered to be the basis of intelligence including emotions. However, there are few studies on the construction of emotional and cognitive models using actual sensory signals since it is difficult to measure these sensory signals continuously. In this study, we capture multimodal sensory signals from a real human body and attempt to integrate and structure this information by applying machine learning methods. Then, we discuss the possibility of reproducing the concepts in the brain by analyzing the integrated structure. In particular, we report on concept formation based on signals obtained in an eating task, and how signals obtained in non-eating tasks are perceived.