Information technologies, such as IoT, artificial intelligence (AI), and virtual reality (VR), have seen so much development that there is now a wide variety of digital equipment incorporated into the infrastructure of daily life. From the agrarian society (Society 1.0) through the information society (Society 4.0), humankind has created farmlands and cities by structuring natural environments physically and has built information environments by structuring them informationally. However, despite the rapid development of information environments, it may be fair to say that the perspectives of the human body have not changed at all since the industrial revolution.

In the context of these recent technological developments, greater attention is being paid to human augmentation studies. These studies aim for a new embodiment of “human-computer integration,” one which can physically and informationally compensate or augment our innate sensory functions, motor functions, and intellectual processing functions by using digital equipment and information systems at will, as if they were our hands and feet. It has also been proposed that the technical systems that enable us to freely do what we want by utilizing human augmentations be called “JIZAI” (freedomization) as opposed to “automation.”

The term “JIZAI body” used in these studies represents the new body image of humans who will utilize engineering and informatics technologies to act at will in the upcoming “super smart society” or “Society 5.0.” In these studies, human augmentation technologies are an important component of JIZAI, but JIZAI is not the same as human augmentation. JIZAI is different in scope from human augmentation, as it aims to enable humans to move freely among the five new human body images: “strengthened sense” (augmented perception), “strengthened physical body” (body augmentation), “separately-designed mind and body” (out of body transform), “shadow cloning,” and “assembling.” In the society of the future where JIZAI bodies widely prevail, we will use technologies that enable us to do what we have failed at or given up due to limitations of our physical bodies. We believe that a future society, one in which aging does not reduce our capabilities but instead increased options give us hope, can be realized. This special issue, consisting of two review papers and twelve research papers, deals with diverse and wide-ranging areas, including human augmentation, robotics, virtual reality, and others. We would like to express our sincere appreciation to all the authors and reviewers of the papers contributed to this special issue and to the editorial committee of the Journal of Robotics and Mechatronics for their gracious cooperation.

Recent advances in human-computer integration (HInt) have focused on the development of human-machine systems, where both human and machine autonomously act upon each other. However, a key challenge in designing such systems is augmenting the user’s physical abilities while maintaining their sense of self-attribution. This challenge is particularly prevalent when both human and machine are capable of acting upon each other, thereby creating a human-machine mutual action (HMMA) system. To address this challenge, we present a design framework that is based on the concept of transparency. We define transparency in HInt as the degree to which users can self-attribute an experience when machines intervene in the users’ action. Using this framework, we form a set of design guidelines and an approach for designing HMMA systems. By using transparency as our focus, we aim to provide a design approach for not only achieving human-machine fusion into a single agent, but also controlling the degrees of fusion at will. This study also highlights the effectiveness of our design approach through an analysis of existing studies that developed HMMA systems. Further development of our design approach is discussed, and future prospects for HInt and HMMA system designs are presented.

In virtual reality (VR), a virtual mirror is often used to display the VR avatar to the user for enhancing the embodiment. The reflected image of the synchronization of the virtual body with the user’s movement is expected to be recognized as the user’s own reflection. In addition to the visuo-motor synchrony, there are some mirror reflection factors that are probably involved in avatar embodiment. This paper reviews literature on the psychological studies that involve mirror-specific self-identification and embodied perception to clarify how the reflected image of the virtual body is embodied. Furthermore, subjective misconceptions about mirror reflections reported in naïve optics have also been reviewed to discuss the potential of virtual mirror displays to modulate avatar embodiment.

Interpersonal communication is so important in everyday life that it is desirable everyone who participates in the conversation is satisfied. However, every participant of the conversation cannot be satisfied in such cases as those wherein only one person cannot keep up with the conversation and feels alienated, or wherein someone cannot communicate non-verbal expressions with his/her conversation partner adequately. In this study, we have focused on facial direction and gaze among the various factors that are said to affect conversational satisfaction. We have attempted to lessen any sense of non-participation in the conversation and increase the conversational satisfaction of the non-participant in a tripartite conversation by modulating the visual information in such a way that the remaining two parties turn toward the non-participating party. In the experiments we have conducted in VR environments, we have reproduced a conversation of two male adults recorded in actual environments using two avatars. The experimental subjects have watched this over their HMDs. The experiments have found that visually modulating the avatars’ faces and gazes such that they appear to turn toward the subjects has increased the subjects’ sense of participation in the conversation. Nevertheless, the experiments have not increased the subjects’ conversational enjoyment, a component of the factors for conversational satisfaction.

People infer the internal characteristics (attitude, intent, thoughts, ability, relationship, etc.) of others (interpersonal cognition, IC) from the impressions they form from the personality or attributes of those others (impression formation). Studies premised on interpersonal communication in a seated condition have confirmed that, regardless of whether the communication is in the real world or in a media environment, the appearance of the other person affects IC and the outcome of the communication. People also develop relationships based on impressions or images of the other party. The psychological relationship manifests in physical relationships, that is, the relative positions of the body or the movement. In this study, we evaluate the effects of the appearance of the opponent’s avatar on the players’ IC in whole-body interaction taking place in a virtual reality (VR) space. Moreover, we examine the feasibility of constructing a method of changing the players’ relationship in interpersonal interactions that accompany the control and interference of the entire body, “whole-body interaction,” by manipulating their appearances. In this study, we selected the party game Twister as a case model of whole-body interaction and developed a system that allows users to play Twister in VR space. Using this system environment, we conducted an experiment to evaluate the players’ IC based on the gender and realism of the opponent’s avatar. The results showed that differences in the appearance of the opponent’s avatar affected the IC of male players. We also indicated that changes in IC observed in the experiment can affect the players’ relationship, thus identifying issues that must be resolved in order to realize the method.

Presenting objects’ texture sensations by electrical stimulation has been drawing greater attention as a means to improve the augmented reality experiences. To reproduce the texture sensations of real objects in detail, tactile perceptions of their complex surface shapes must be translated as electrical stimulation. Many conventional studies on presenting vibrational sensations by electrical stimulations have used single pulse waveforms, limiting the vibrational sensations that can be presented by electrical stimulations. In this paper, therefore, we propose the local peak method, in which pulse waveforms are configured on the objects’ surface structures. We have experimentally proved that the proposed local peak method can present sensations that are more akin to touching physical objects than pulse waveforms of single intervals in the case of presented objects having complex surface structures.

Menstrual symptoms and cycles are complex, and the associated discomfort is difficult to quantify. Therefore, men, and some women, do not completely understand them. Here, we propose a system that simulates menstruation-like cramps through electrical muscle stimulation (EMS). We conducted an experiment to compare and evaluate the natural and electrically stimulated menstrual cramps. The results show that menstrual cramps using EMS can reproduce the nature of periodic dull pain. However, in this study, the position where the pain occurred was shallow. Furthermore, we constructed a demonstration system based on the proposed method. From the exhibition, we confirmed that this experience can help verbalize menstrual-related discomfort and allow people to better understand menstrual symptoms. In other words, this experience will help eliminate negative perception of menstruation.

Deep robotic learning by learning from demonstration allows robots to mimic a given demonstration and generalize their performance to unknown task setups. However, this generalization ability is heavily affected by the number of demonstrations, which can be costly to manually generate. Without sufficient demonstrations, robots tend to overfit to the available demonstrations and lose the robustness offered by deep learning. Applying the concept of motor babbling – a process similar to that by which human infants move their bodies randomly to obtain proprioception – is also effective for allowing robots to enhance their generalization ability. Furthermore, the generation of babbling data is simpler than task-oriented demonstrations. Previous researches use motor babbling in the concept of pre-training and fine-tuning but have the problem of the babbling data being overwritten by the task data. In this work, we propose an RNN-based robot-control framework capable of leveraging targetless babbling data to aid the robot in acquiring proprioception and increasing the generalization ability of the learned task data by learning both babbling and task data simultaneously. Through simultaneous learning, our framework can use the dynamics obtained from babbling data to learn the target task efficiently. In the experiment, we prepare demonstrations of a block-picking task and aimless-babbling data. With our framework, the robot can learn tasks faster and show greater generalization ability when blocks are at unknown positions or move during execution.

In recent years, it has become possible to experience sports in the virtual reality (VR) space. Although many haptic displays in the VR environment currently use vibrators as the mainstream, the vibrators’ presentation is not suitable to express ball-receiving in the VR sports experience. Therefore, we have developed a novel haptic display that reproduces an impulsive force by instantaneously applying traction to the palm using a string and wearable brake system. This paper proposes a method to present various reaction forces by dynamic control of the braking system and report the quantitative evaluation of the device’s physical and psychological usability.

In this paper, we propose a method for recognizing handwritten characters by a finger using acoustic signals. This method is carried out using a smartphone placed on a flat surface, such as a desk. Specifically, this method uses an ultrasonic wave transmitted from the smartphone, which is reflected by the finger, and an audible sound is generated when writing a handwritten character. The proposed method does not require an additional device for handwritten character recognition because it uses the microphone/speaker built into the device. Evaluation results showed that it was able to recognize 36 types of characters with an average accuracy of 77.8% in a low noise environment for 10 subjects. In addition, it was verified that combining an audible sound and an ultrasonic wave in this method achieved higher recognition accuracy than when only an audible sound or an ultrasonic wave was used.

A patient who unexpectedly sustains a lower leg injury often uses crutches for walking, to both assist the patient and prevent further injury until a full recovery is made. In this study, a novel crutch walk training system was developed; a head-mounted display (HMD) was introduced to present a virtual obstacle for trainees using crutches. Through repetitions of the training, it was expected that the trainees would improve their overall body stability. Healthy subjects participated in experimental training exercises using the developed training system. As a result, the participants acquired the skill of walking well with crutches without losing their body balance significantly, even in the event of facing a virtual obstacle, during the five training sessions.

Tactile sensations are crucial for achieving precise operations. A haptic connection between a human operator and a robot has the potential to promote smooth human-robot collaboration (HRC). In this study, we assemble a bilaterally shared haptic system for grasping operations, such as both hands of humans using a bottle cap-opening task. A robot arm controls the grasping force according to the tactile information from the human that opens the cap with a finger-attached acceleration sensor. Then, the grasping force of the robot arm is fed back to the human using a wearable squeezing display. Three experiments are conducted: measurement of the just noticeable difference in the tactile display, a collaborative task with different bottles under two conditions, with and without tactile feedback, including psychological evaluations using a questionnaire, and a collaborative task under an explicit strategy. The results obtained showed that the tactile feedback provided the confidence that the cooperative robot was adjusting its action and improved the stability of the task with the explicit strategy. The results indicate the effectiveness of the tactile feedback and the requirement for an explicit strategy of operators, providing insight into the design of an HRC with bilaterally shared haptic perception.

The thermosensory system may misidentify a temperature stimulus with different thermal properties. The mechanism of this hot-cold confusion has not been clarified; hence, it has not yet been applied. In this study, we created a wearable temperature presentation device that is closer to the application and analyzed the tendency and mechanism of temperature confusion by analyzing the hot-cold confusion of temperature sensation in the fingers, which are most frequently in contact with objects. Two experiments were performed. In the first experiment, we presented stimuli on the tips of three fingers (first, second, and third fingers). In the second experiment, we presented stimuli at the center of the distal phalanx, middle phalanx, and proximal phalanx of the first finger. The experimental results indicated the occurrence of hot-cold confusion. Domination, in which the center is dominated by both ends, and a mutual effect, in which the center interacts with both ends, were observed.

Galvanic taste stimulation (GTS) is a non-invasive electrical stimulation of sensory nerves that induces, inhibits, and enhances taste sensation. It has been shown that the cathodal GTS taste enhancement effect occurs when only cathodal electrodes are attached in or near the mouth, while anodal GTS, whose anodal electrodes are attached in or near the mouth, induces an electrical taste sensation rather than taste enhancement. In the present study, we focused on the taste enhancement effect of anodal GTS, enhancing the salty taste produced by a sodium chloride (NaCl) aqueous solution during stimulation. In this study, GTS was applied to the chin rather than the inner mouth so as not to disturb natural eating and drinking behavior, according to a previous study. To demonstrate and quantitatively evaluate its enhancement effect, we conducted two psychophysical experiments in which subjects were asked to indicate the intensity of the saltiness perceived during electrical stimulation by adjusting the concentration of aqueous NaCl to achieve a solution of equivalent saltiness. We discovered that the perceived intensity increased as the current intensity applied to the chin increased. Moreover, the magnification ratios against the baseline exceeded 1 under all conditions of the NaCl aqueous solution. These results indicate that anodal GTS is effective in enhancing the salty taste of NaCl aqueous solutions.

In this paper, we report on the “VIsual Dataset for Visually Impaired Persons” (VIDVIP), a dataset for obstacle detection during sidewalk travel. In recent years, there have been many reports on assistive technologies using deep learning and computer vision technologies; nevertheless, developers cannot implement the corresponding applications without datasets. Although a number of open-source datasets have been released by research institutes and companies, large-scale datasets are not as abundant in the field of disability support, owing to their high development costs. Therefore, we began developing a dataset for outdoor mobility support for the visually impaired in April 2018. As of May 1, 2021, we have annotated 538,747 instances for 32,036 images in 39 classes of labels. We have implemented and tested navigation systems and other applications that utilize our dataset. In this study, we first compare our dataset with other general-purpose datasets, and show that our dataset has properties similar to those of datasets for automated driving. As a result of the discussion on the characteristics of the dataset, it is shown that the nature of the image shooting location, rather than the regional characteristics, tends to affect the annotation ratio. Accordingly, it is possible to examine the type of location based on the nature of the shooting location, and to infer the maintenance statuses of traffic facilities (such as Braille blocks) from the annotation ratio.

In schools and other educational institutions, there are many instances that require the arrangement of chairs depending on the required purpose, such as a class or event. Some chairs are on casters; however, educational institutions typically use stacking or folding chairs. Although effective in terms of storage, these must be lifted by hand during transportation, increasing the burden on the workforce. While automation of baggage transport in warehouses has improved significantly, little attention has been paid to the automation of chair transport. Despite the demand and the fact that self-propelled chairs have already been developed, automatic transport of chairs without casters has never been reported. In this study, we constructed an automatic chair-transport system using an omnidirectional mobile robot and focused on a stacking chair that allowed the robot to position itself underneath. The developed system utilizes the image of the seat and frame pipe of the stacking chair to estimate the chair’s position and direction with respect to the robot. Once the robot has positioned itself under the chair, the chair is lifted and transported using a lifter device attached to the robot.

In recent years, there has been an increasing need for larger screens and higher definition displays, while projectors are becoming smaller and cheaper. Furthermore, an ultra-short-throw projector that can display on a large screen while significantly reducing the distance between the projector and screen is being developed. However, ultra-short-throw projectors are required to be precisely aligned with the screen, and if the screen is not flat, the projected image becomes distorted. Therefore, geometric correction projection technology is attracting attention for projection on curtains and the walls of living rooms instead of screens for realizing the correction of distortion during projection with ultra-short-throw projectors, projection mapping, signage, etc. We focused on developing a hologram with perfect command of the ray. Conventional geometry-correction systems are expensive systems that require a personal computer and a camera. In this study, we developed a geometric correction method applying holographic ray direction control technology to control a holographic ray at a low cost and in real time. In this paper, we studied the exposure technology and proposed a ray-direction control technology that combines a scanning laser projector that uses a hologram and a micro electro mechanical systems mirror. We also proposed and demonstrated the basic principle of a holographic surface projector (HSP), which uses hologram geometry correction technology. Finally, we constructed a geometrically corrected hologram exposure system using a depth camera and conducted geometrically corrected projection experiments.

In ultra-compact electric vehicles, the satisfactory installation of an assist mechanism for steering operation is difficult. To address this problem, in this paper, we propose an active steering wheel system in which the steering wheel and tires are electrically connected, without a mechanical connection. Furthermore, in ultra-compact mobility vehicles where the driving position is restricted, steering burden is likely to occur depending on the physique of the driver. However, whether the effects of the steering reaction torque and the amount of steering increase the burden on the driver in such vehicles has not yet been clarified. Therefore, in this study, we developed an upper limb burden model using inverse kinematics and muscle activity to investigate the burden of steering on the driver by considering the driver physique.

We are developing a small compact electric vehicle for shopping purposes. In this study, we fabricated an experimental vehicle, that uses only six small rechargeable AA batteries (7.2 V, approximately 2 A·h) as an electric power source. The vehicle user can select between two driving positions: standing and sitting. A compact transmission with a 90-W DC motor and a speed controller that uses pulse width modulation control was designed as an actuating system. Running experiments were conducted to observe the performance of the fabricated vehicle on a flat floor in a gymnasium. The fabricated vehicle was able to operate for 52 min at a speed of 2.73 km/h. The getting-on and getting-off processes in the vehicle were repeated many times during shopping. The human leg strain while getting on and off the vehicle was investigated by electromyogram measurement. During the getting-on and getting-off processes in the vehicle, the myoelectric potential of the quadriceps increased in the sitting position but did not increase in the standing position. The experimental results show that a user suffers more strain in the sitting position than in the standing position.

Robots are required to be significantly compliant and versatile to work in unstructured environments. In a number of studies, robots have positively exploited the environments during interactions and completed tasks from a morphological viewpoint. Modular robots can help realize real-world adaptive robots. Researchers have been investigating the actuation, coupling, and communication mechanisms among these robots to realize versatility. However, the diverse force transmission among modules needs to be further studied to achieve the adaptive whole-body dynamics of a robot. In this study, we fabricated a modular robot and proposed the realization of force transmission on this robot, by constructing fluid transferable network systems on the actuation modules. By exploiting the physical property variations of the modular robot, our experimental results prove that the robot’s motion can be changed by switching the connection pattern of the system.