We have developed a novel sensation interface using galvanic vestibular stimulation (GVS). The vestibular system is stimulated by weak current through the electrodes, placed behind the ears. GVS causes lateral virtual acceleration toward the anode, which shifts the sense of balance. The GVS interface can induce lateral walking diverging from intended straight line. Based on this GVS interface technology, we produced an artwork on the subject of wavering identity in the modern society. In our artwork, the compact display is floating on the water. An acceleration sensor is integrated into the display, and the obtained data is sent to the GVS interface. GVS is presented according to the data form the sensor. Any kind of vibration of the display disturbs the balance of the wearers. When the display falls over, they feel big swaying sensation. This GVS interaction makes them feel truly connected to the display. They keep on walking, while holding the tank of water. This artwork is intended to observe and hold your wavering identity (the display on the water) from the outer perspective.

View full abstract

The virtual reality has made it possible for us to experience nonexistent environments, or even fictional worlds. It is also useful for creating artistic effects, especially which apply to multiple human senses. This paper reports the installation work 'Majestic Imperator' inspired from the transportation facilities for exclusive royal families or nobilities in the end of modern times. In this work, some virtual reality techniques are used for representing the "phenomena" which were experienced by them lived in the old times. To promote the reality, four kinds of human senses are stimulated in the same time, by being given dynamic vibration from the floor, wind pressure and so on.

View full abstract

We anticipate that employing an enjoyable user interface will excite the user's interest thus enhancing their motivation. Using virtual reality, which is capable of producing effects that cannot occur in the real world, we developed an optical-illusion-like interface for block-based 3D modeling software. Based on this idea, we developed a block-based modeling software called Incompatible BLOCK that has an interface that allows the user to position cubes efficiently as well as enjoyably. In this paper, we describe this interface and the algorithm of Incompatible BLOCK and discuss the effectiveness of the proposed interface based on the reactions of those who used Incompatible BLOCK.

View full abstract

In this paper, we report about a creation of art that consists of two ferrofluid sculptures that moves synthetically to music. The two spiral towers stand on a large plate that hold ferrofluid. The body of the tower was made by a new technique called "ferrofluid sculpture" invented by Kodama, which enables artists to create dynamic sculptures with fluid materials. This technique uses one electromagnet, and its iron core is extended and sculpted. The ferrofluid covers the sculpted surface of a three-dimensional iron shape that was made on an electronic NC lathe. The movement of the spikes in the fluid is controlled dynamically on the surface by adjusting the power of the electromagnet. In this work, we tried to activate fluid precisely by utilizing digital music metadata. To control the synchronization of the ferrofluid with the music playback in real time, time series metadata are added to the music beforehand. The metadata consist of musical information, such as beat position, chord progression, and melody block information, and ferrofluid control information such as DC bias voltage and AC pattern. Each data record has a time stamp that indicates the timing of presentation. All data are stored in time-series order. These time series metadata must be accurate for precise control of timing, so as to cancel the time delay of fluid movement. By this correction, the time when the protuberance of the spike reaches its maximum size is coincident with the beat of the music. As a result, the rhythm of the fluid movement coincides with the musical rhythm.

View full abstract

We developed a multi-user Mixed Reality (MR) attraction "Watch the Birdie!" as a test bed of an MR system with new modalities such as Audio-Visual MR and a sound event detection technique. In this attraction, users can find the approaching birds not only by CG but also by the loudness and direction of the chirp sounds from a three-dimensional sound field system. Users can also interact with birds by using sound direction and position. The sound detection system can localize the position of the sound source made by each user, with one pair of microphone array. Additionally, in this attraction, there are other functions that bush rustles when the birds appear from there and a user can feel the weight of the bird when it perches on his/her hand.

View full abstract

"Three men, three legs" is a work whereby three users cooperate within a prepared three-dimensional CG cyberspace. In the space, an avatar (virtual body) is provided with three legs and one spherical body; through the network an interactive environment is created where three people share this body. One person operates each leg, and each user can move the leg they are operating freely, but cannot control other users' legs. To manage the legs of the avatar three interfaces were prepared and the differences of each interface were investigated. In particular, haptic devices were found suitable to display sensations of sharing a body. I clarify the production process and concept of this work, and propose a three-person interactive network application. In addition, I discuss the users' change in thought process resulting from sharing one body through this work, and indicate future prospects.

View full abstract

While air surrounds our body, we rarely see its flow because the air is invisible. This paper discusses a new method for visible recognition of wind. When the users of this simulation system draw whatever objects they might want to see the air flow around them, the system executes real-time 2D computational fluid dynamics around those objects and renders the result over the drawn object. Main purpose of the system is that it will make easier for many people to recognize wind and play with the simulated wind. This method may be applied to many areas such as media art, education, designing or entertainment.

View full abstract

A projector mounted on mobile devices, which a user can freely move in a 3D space, could be not only an output device but also an input device which allows humans to manipulate mobile robots. Therefore, we explore human robot interaction using a handheld projector. For this purpose, we developed a cooperative game, CoGAME, in which players guide a simple robot with handheld projectors. In this game, players can connect projected maps each other, and they lead a robot to the goal cooperatively. In this paper, we present some techniques used in CoGAME, and show its evaluation.

View full abstract

This paper describes Sound Scope Headphones, which enable the user to control an audio mixer through natural head and hand movements. When listening to music, a listener might want to hear particular parts more clearly or reduce the level of some parts. Such a capability is possible with commercial audio mixers that perform multi-channel recording and mix-downs. However, commercial audio mixers are too complicated for musical novices to be able to control numerous channel volumes and panpots. Our headphone device controls an audio mixer by using three sensors: a digital compass, a tilt sensor, and a distance sensor. These sensors are mounted on the headphones and they detect natural movements, such as of the head or a hand placed behind an ear when the user is listening to music. By putting their hand behind their ear, the listener adjusts the distance sensor on the headphones to focus on a particular part that they want to hear. We made three distance sensor prototypes and experimental results show that musical novices are more able to find a target instrument in a multi-channel recording with an infrared distance sensor.

View full abstract

We propose a mixed reality application that uses a humanoid robot as a CG character's tangible body. The system consists of a synchronized pair of a humanoid robot and a virtual avatar, and an HMD that overlays the avatar onto the robot. And this research experiments the evaluation of the present enhancement of CG character through physical interaction with a robot which the character is overlaid onto it. The application requires that the motion data should be the processed motion-capture data to enhance its reality, the robot should be light-weight and have flexible link arms to make it safe.

View full abstract

In this paper, we developed an immersive 3D wind display as a method for extending virtual reality. In addition, we introduced applications for the system; development of 3D wind camera and evaluation of the contents, an application for video game interfaces and media arts, and an application for image synthesis used in combination with image based lighting.

View full abstract

This paper proposes a new display system for creating Mixed Reality environment. This system employs two imaging optics for displaying two real images; one is upright (vertical) and the other is flat (horizontal). In general, we feel something strange to see a real image formed by lens optics. To solve this problem, some additional information is very helpful. In our system, the horizontal real image is used for enhancing the reality and the depth feel of the vertical real image. Experimental results show the potential applicability of our system as a Mixed Reality display system without glasses to wear.

View full abstract

Due to recent innovations on location-tracking technologies and motion-sensing technologies, it is possible to store our locomotion in daily lives. From the view point of contents creation and authoring, those technologies can be used like motion-capture system which is applied to digital cinema, game, and 3D-CG. In this paper, we describe the basic concept of locomotion-capturing and its applications toward art and entertainment. Our system called "GeoWalker" is the versatile creative environment for converting logged data into various expressions such as animation, drawing, sound, and dancing. Finally, we clarify the potentiality and possibility of locomotion-capturing contents.

View full abstract

In the pre-production process of filmmaking, PreViz, previsualizing the desired scene by CGI, is used as a new technique. In an alternative approach, mixed reality (MR) merges the real and virtual worlds to achieve real-time interaction between users and MR space. We propose MR-PreViz that utilized MR technology in PreViz. MR-PreViz makes it possible to merge the real background and the human and creature by CGI in open set and outdoor location. The user can consider the camera-work and camera blocking efficiently by using MR-PreViz. This paper introduces the basic concept of MR-PreViz, the design of main system, the software architecture, an authoring tool supporting selection of camera-work, and CWML (Camera-Work Markup Language) to describe camera-work.

View full abstract

We have developed and evaluated a tabletop virtual Ohajiki game using a finger flicking gesture recognition system called "Ohajiki Interface". Ohajiki is a Japanese traditional marble game. A player flicks a flat marble that is called "Ohajiki" on the table with his finger. This system can recognize a user's hand position and its finger speed using a very rapid image recognition, and enables a user to flick a virtual Ohajiki projected on a table with his finger.

View full abstract

Recently on the Internet, with such phenomena as Consumer Generated Media and mash-ups, various derivative creation services for a diverse range of modalities have appeared. However, even in the midst of massive growth of contents resulting from these derivative creations, individual service providers have not yet started to collaborate beyond corporate borders in order to attain multi-derivative creations. The authors implemented and evaluated the HIVE model that leads to multi-derivative creation through exploitation of digital rights licensing, from the viewpoint of Media Art Archives.

View full abstract

Existing interactive dramas adapt their stories based on user interactions such as actions and dialogues. In our research we aim at incorporating user bio signals into such systems. In this paper, we focus on a particular drama genre called suspense, where the viewer develops suspense emotion through expectation, curiosity, or surprise of the result to the hero or heroine in the scene. In particular, we propose a method for classification of suspense emotion with dynamic time warping from galvanic skin responses. Representative signals for each suspense type are discussed. In addition, the classification performance of the proposed method is examined and compared with a brute-force method. Evaluation results confirm the effectiveness of the proposed method as a good candidate for being implemented in a real interactive drama system.

View full abstract

We developed position-dependent visible light communication using Digital Micromirror Device (DMD), which can display information for human and bit for sensors at the same time and the same position. In this paper, we indicate the purposes and relations between the our system and the other studies of ProCam at first. Secondly, we describe the basic idea how to communicate with visible light that depends on position. Thirdly, we developed an algorithm of embedding bit signals to control machines into static image such as a photo. This algorithm uses pulse width modulation (PWM) to express the tone of color with digital signals. These ideas are demonstrated using DMD that acts as the switch of light. We implemented the projector using DMD, and test the position dependent communication by visible light. Finally, as an application of this system, we implement a receiver and small display (8x8 LED dot matrix) module that can receive and decode the received projection light. As a result, we made sure that the position dependent visible light communication with DMD is suitable for the art and entertainment system.

View full abstract

To draw pictures is creative activity, and one of effective method of self-expression. Generally speaking, however, as people get older, the opportunities to draw pictures decrease and people become not to enjoy drawing in the daily life. Therefore, our project aims to bring drawing, which is essentially enjoyable and makes the life creative, closer to the public people. In this paper, the authors propose "Minimal Drawing", which can solve the problems mentioned above. In order to achieve minimal drawing experience, the authors developed a drawing environment, called as "Roll Canvas". It has the circulated canvas. The user can control the rotation of the canvas, and he/she can draw any kind of pictures freely when the canvas is rotating. Unexpected line drawing can be generated due to the contingency between user's drawing action and the rotation of the canvas. The drawing sometimes can be an artistic picture or a textile design. In this way, the minimal drawing experience includes both user's intension and accidental strokes, which can open up further possibility of drawing to the wide range of people.

View full abstract

Ototenji is a braille learning device for people who have the ability to see. The device has a practical use, and has been presented in various locations as art and as entertainment. As a result, its presentations as an artwork have lead to heightened interest among those with eyesight. This introduction will review efforts in using this device to lead toward greater understanding about visual impairment.

View full abstract

This paper describes the development of the display that controls the light and shadow cast on the floor using illumination lamp or natural sunlight as a light source. Polymer Dispersed Liquid Crystals (PDLC) film that can switch from opaque to transparent rapidly is used in each pixel of the display. The display is intended to install in ceiling or window and artificially generates images of shadow on the ground or wall just like sunshine filtering through foliage. Our concept is to develop the display that can appeal to person's instinct as an interactive art installation.

View full abstract

We present a communication interface that achieves handsfree conversation in the sight-distance. The wearer of this interface can hear the voices only from the direction in which he/she is facing, and can send his/her voices only in the direction towards which he/she is facing. Therefore, individuals who are within sight of each other can have a conversation, even if they are not close enough to talk directory. This interface promotes communication between people who are within sight of each other, using the communication protocol in the personal-distance. In this paper, we describe the design theory of this interface and its applications.

View full abstract

In this paper, we propose a concept of ambient online-game system. The system is composed with integrating of environment information and dynamic episode control. The sensory information such as temperature, humidity and illuminance are applied to the trigger of scenario control. The scenario controller generates various story developments having story data consisting of the episode hierarchy structure. This hierarchy structure is classified in "reaction" "story" "plans" and can express the various actions of the character. This paper describes also that we constituted a part of "the breeding game" with these technology.

View full abstract

In this paper, we discuss application possibilities of force perception technologies for hand-held devices in the field of entertainment. We developed an interactive system with the force perception technology called "Come Over Here, or Catch You!", which consists of a hand-held haptic interface and a position and posture identification system. Since the hand-held haptic interface does not require an external ground, it can be used outside the laboratories and does not interrupt human's behavior. We verified the feasibility of the system through results of questionnaires by the experienced.

View full abstract

Recently we have developed a novel stage design system, which can present different vibro-auditory information to each member of audience. The system is composed of the ultrasonic speaker and the moving controller. The speaker delivers ultrasonic wave, which is modulated with the audible frequency. The wave travels straight like a beam, and when it encounters an object, the object is vibrated with the audible modulation frequency. The speaker, therefore, can directly and selectively present sounds to the audience, and it can generate a virtual sound source anywhere in the installation space such as a wall, a screen, or even performer's body. Although the speaker is generally fixed and the direction of the "sound spot" is limited, we attached the ultrasonic vibration component of the speaker to the top of a moving light housing to control the direction of the sound beam in real time. This Moving Ultrasonic Speaker (MUS) was used in the multi-media performance entitled "Living Lens" on 15, 16, 17 July 2006, as a part of Brisbane Festival 2006. In this paper, we give an outline of this stage design technique, and propose a concept of "Vibro-scape Design" in the performing art stages.

View full abstract

Recently there has been an amount of application of Virtual Reality. However, there may exist risk of health hazards in virtual space. There reasons remain unsolved and there measures are not established yet. Then we began to examine the impression from objects in virtual space. From the experiments on the impression of walls with various patterns in virtual spaces, the results show impressions were affected by the patterns of walls.

View full abstract

This paper introduces a music interactive system that aims to provide the users with the experience of creating rhythm freely and to maintain the tempo as a part of music edutainment experience. Our systems algorithm provides the user with rhythmic hint and melodic hint as ways to give feedback to the users. We describe the system setup, an algorithm for recognizing inputted tempo and user studies that evaluate the effectiveness of our system.

View full abstract

As a display for displaying three-dimensional VR objects in large spaces such as a public space, we have proposed "Controllable particle display". Also, as a "Controllable particle display", we have proposed a system of using water drops. In this system, a plane surface of water drops falls down and images are projected upward according to the position of the water plane. Therefore, stacking images creates a volume. Based on this method, we implemented a prototype system whose display area is 72[mm] × 72[mm] × 309[mm], and exhibited the prototype system. From the results of these experiments, we demonstrated the effectiveness of our concept.

View full abstract

Various forms of art and entertainment involve many different characters, and advances in human interfaces have necessitated physical interactions in order to develop an improved sense of reality. In this paper we propose a method for generating the motions of characters using multidimensional keyframe animation in parallel with real-time physical simulation. The method generates characters capable of physical interaction, and also allows animators to use traditional methods for designing character motion. We have applied the method to a project entitled "Kobito -Virtual Brownies-" and confirmed its effectiveness experimentally.

View full abstract