Much attention has been paid recently to Augmented Reality (AR) which supports human's daily life by blending multimodal information into real world, typically on the objects being considered. Most of existing AR systems provide such information to human by using ordinary displays and see-through head mounted displays. However, monitoring and understanding of human actions is necessary for the system to achieve flexible and gentle interactions. The paper presents an attempt to make such interface using an extended digital desk for guiding and teaching robot system. A prototype system consists of a projector subsystem for information display and a realtime tracking vision subsystem for recognizing human's action. We implemented coordination of two levels of interactions using a virtual teaching panel and projected real image of the task environment on a desk for intuitive teaching. Experiments of teaching robot tasks demonstrate usefulness of the proposed system.