We are developing daily life-support robot ASAHI 2013, which is easily operated by the user using verbal communications and gesture. The spoken dialog is the most natural way of giving directions to the robot; however, it is difficult to deal with the directions containing demonstrative pronouns when using only verbal information. To solve this problem, we have proposed a method that uses spoken dialog and pointing gesture. We used RGB-D sensor for recognizing the gesture and detecting the object on the floor. From the experimental result, we confirmed that the distances from the robot to the target points have large error, while the angles of the points were relatively accurate. Thus, the proposal method controls a robot using only the angle information. Next, the RGB-D sensor detects the object on the floor. This paper describes the method to fetch the object based on the pointing gesture and spoken dialog.
