An Acquisition System of Concept and Grammar Based on Combining with Visual and Auditory Information

Abstract

Human beings accept external stimuli through the senses, and the information is transmitted to the brain. It seems that human beings acquire various concepts from these information. Particularly, auditory and vision stimuli play an important role in the concept acquisition. From this point of view, we developed a system which acquired concepts and grammar from visual and acoustic information without a priori knowledge by comparing input information with acquired concepts. And, the system generates sentences (or speech) for input images and image concepts for input sentence. A concept consists of a set of relation between a figure feature and a speech event. A figure consists of four categories i. e. shape, size, color and position, and speech events consist of speech waves which are extracted from input speech. If the system perceives the acquired concepts in the input speech, the learning process begins from mapping between the input information and the perceived concept. The part of input which was not spotted by acquired concepts is registered as the new concept in the concept dictionary. And, our system perceives the sequence of concept inputs by using acquired concepts, and models these sequences by using HMM, in other words, it acquires the grammar. Furthermore, our system generates an utterance which explains input images by using acquired concepts and grammar. Our system acquired 11out of 12 types of concepts from 100 pairs of utterance and images. Using the left-to-right HMM for grammar acquisition, the probability our system will generate correct sentences for input images is about 50%. We have realized the first stage of human's concept acquisition process on a computer system.