Understanding Natural Language Instructions for Self-Driving Cars and Grounding

Abstract

Recently, practical applications of automatic driving have been rapidly developing. In the future, it must be necessary to enable interactive operation by natural language in order to easily operate autonomous cars. We therefore attempt to realize the correspondence relationship, i.e., a part of symbol grounding, between the driving instructions expressed in natural language and the objects in the real world recognized by the sensors equipped with a car, and then convert the driving instructions into the particular spatial meaning description to operate autonomous cars. In this study, we particularly focus on the parking operation of a car. We propose two methods: one is extracting spatial semantics from parking instructions, and the other is corresponding spatial semantics with the real-world environment. The structure trees given by Combinatory Categorial Grammar (CCG) are used as intermediate representation in exacting spatial semantics. If unknown words appear, we estimate them by using Conditional Random Field. In order to increase the accuracy of CCG parser, we implement a reranker of parse trees. These parse trees are converted into tree structures called Spatial Description Clause (SDC). We extend the framework of SDC by adding two new semantic categories, VIEW and STATE, so as to be able to ground more variety of the instructions for driving a car in the real-world environment. In corresponding spatial semantics with the real-world environment, we generate probability graphical models called Generalized Grounding Graph and output places or objects which correspond each word. The accuracy of all grounding among the sentences correctly parsed is 79.2%.