Building the Multi Modal Presentation System for an Electrical Device's Operation

Abstract

A multimodal presentation system is proposed, which explains how to operate an electrical device to realize a function of the devices. The explanation should be based on a sequence of operations from an initial state to a desired state. So, NOAH, the hierarchical planner that resolve conflicts between opetators, is used to generate an operator network that describe a process to obtain the desired state of the device. Explanations show each operator or state that compose the network with text and graphics. The operator connects a given state and an obtain state. Problems to translate a operator network to text with graphics are as follows: 1) which should be presented, operator itself, the given state, or the obtained state, 2) which should be presented by text, and 3) which should be presented by graphics. 6 paterns are defined as Describe Action (DA) to generate an expression of elements of the network. DAs are translated from the operator network, then the expression is generated using the suitable modalities. The multimodal presentation system consists of a controller, a planner, an analyzer and a generator. The controller manages an input from user, and makes the desired state. The planner generates an operator network to get the desired state. On the analyzer, the operator network is analyzed and translated to DAs. The generator receives DAs and generates the explanation with text and graphics.