In recent years, advances in CMOS imagers and AI have rapidly expanded the application fields of image processing. Spatial resolution, sensitivity, dynamic range, etc. for devices have dramatically improved, while various recognition functions have been implemented with the introduction of learning-based information processing, making progress day by day. However, the temporal resolution or temporal dynamics of an image has been limited to the video rate level of processing, because image processing has been required to realize the functions of the human eye. In the case of processing for high-speed moving objects or controlling machine dynamics as machine eyes, rather than processing in the range visible to the human eye, high-speed vision, i.e., high-speed image processing in a bandwidth that covers the dynamics of the object, and a system that utilizes such processing, are required. This special issue summarizes such advanced research on high-speed vision, highlighting its current status and future development in the areas of devices, systems, and application developments.

High-speed vision has entered a new era, as the basic technology and various applications have been developed and new functions are being added one after another. We hope that this timely special issue will help its readers grasp this major technological trend and create new system values.

Currently, high-speed vision based on parallel processing exists, and its various applications as high-speed intelligent systems have been proposed and implemented. The basic goal of high-speed vision is to realize vision capabilities and systems that operate at speeds necessary for intelligent systems, in which intelligence operating at the speed inherently required by the application system is achieved. This paper described the vision chip and parallel image processing architectures, presented outlines of system architectures, image-processing algorithms, and related peripheral technologies; described the concepts required to configure high-speed intelligent systems, such as hierarchical parallel distributed architecture, parallel decomposition, orthogonal decomposition, dynamics matching, latency minimization, high-speed 3D shape measurement, active vision, tracking vision, dynamic compensation, and dynamic projection mapping; and discussed a wide range of application systems in a systematic manner.

In this study, we demonstrate the implementation of make-to-order bead art assembly without human intervention using dynamic compensation approach to achieve accurate real-time positioning and long-term adaptation for robotic automation in smart manufacturing. In the proposed framework, an industrial robot was designed to perform coarse global motion to implement low-bandwidth adaptation. Simultaneously, fine local motion to tackle real-time online uncertainties was achieved using an add-on robotic module to implement accurate positioning. The effectiveness of the proposed method was verified through experimental evaluations.

This study focuses on robotic assistance for peg-and-hole alignment with micrometer-order clearance. The objective of the robotic assistance is to cooperate with a human operator based on a coarse-to-fine strategy in which the human operator conducts coarse alignment and the robotic assistance realizes fine alignment. Robotic-assisted fine alignment is achieved by mimicking the process toward annularity of an annular solar eclipse. The first principal axis of a specified image feature (we call it a eclipse feature) is calculated by subtracting the surfaces of a hole part (a small gear with an inner diameter of 1 mm) and a peg part (a shaft with a diameter of 0.95 mm). Accordingly, control strategy is developed to realize accurate alignment. Moreover, the effectiveness of the proposed method is verified by experimental evaluation.

Our research group has developed an air hockey robot with high-speed vision capable of visual recognition at 500 Hz, which exceeds that of humans. Although this robot uses high-speed vision to increase its reaction speed and improve its performance, it has a low speed to compete with humans because its hand speed is still not sufficiently fast. In this study, a high-speed wrist rotation mechanism using a direct drive motor was introduced to the air hockey robot to improve the hitting speed through the snapping motion of the wrist. Furthermore, we developed a hitting motion algorithm that utilizes high-speed visual feedback. The effectiveness of the proposed system was verified experimentally. The proposed system exhibits an improved motion performance.

Making robots capable of pouring can be useful in both service and industrial applications. Considering the importance of controlling liquid vibration in mixing chemical reagents and other industrial applications, we investigated in the this study robotic pouring with the aim of controlling liquid vibration, more specifically, the beer-foam ratio during beer pouring. We propose a vision-based measurement method that can measure the liquid volume with an error of less than 5% in real time. Besides, together with a proposed robot pouring controller, we develop a robot pouring system that can control ratio of beer-foam volume with an error of less than 5% during pouring. The flexibility of the developed system was also demonstrated through experiments using different types of container and beer.

In this paper, we propose a measurement system that employs a camera array based on a non-telecentric optical system and an accompanying measurement algorithm to measure the straightness, length, diameter, and appearance of a rod. Measurements using telecentric optical systems, which employ orthogonal projection to preserve the dimensional ratios regardless of distance, are common in image-based inspection of the dimensional or geometrical tolerances of industrial products. However, some cases depend on the size of the target or inspection item, wherein it is difficult to configure a measurement system using telecentric optical systems. As an example, this study considers the measurement of straightness of a long rod and shows that it is possible to achieve high measurement accuracy using non-telecentric optical systems by introducing methods to calibrate miniscule errors and distortions that remain uncorrected in conventional image calibration methods. We also show that the same measurement allows for the measurement of other inspection items and evaluation of their respective measurement accuracies, thereby proving that a flexible image-based inspection process can be constructed using the proposed system and method.

Efficient imaging is achieved under conditions of high relative velocity between the camera and the subject by using the following imaging system; two galvanometer mirrors are placed vertically in front of the camera, one for motion blur compensation and the other for switching the angle of view. The proposed system can overcome the shortcomings of conventional imaging systems with motion blur compensation, such as a small angle of view, and efficiently acquire high-resolution images. If the angle is changed for each capture while the mirrors are stationary for the exposure time, the natural frequency of the mirrors produces noise, leading to a poor resolution. However, this issue can be managed by generating and using an input that does not contain a natural frequency component. A target moving in one dimension can be captured and it is confirmed that the angle of view was extended from the obtained image. It is expected that the camera will be used for inspections under conditions where the relative speed between the camera and target is high, such as in highway tunnel inspections.

Vehicle-mounted inspection systems have been developed to perform daily inspections of tunnel lining. However, it is difficult to continuously scan images at the desired target angle with the camera’s limited visual field because the vehicle does not always maintain the same position in the car lane while it runs. In this study, we propose a calibration-free optical self-localization method based on white line recognition, which can stabilize the scanning angle and thus enable the scanning of the entire tunnel lining surface. Self-localization based on the present method has a processing time of 27.3 ms, making it valid for real-time view angle compensation inside tunnels, where global positioning systems cannot operate. It is robust against changes in brightness at tunnel entrances and exits and capable of dealing with car lanes with dotted lines and regular white lines. We confirmed that it is possible to perform self-localization with 97.4% accuracy using the scanned data of white lines captured while the vehicle is running. When this self-localization was combined with an inspection system capable of motion blur correction, we were able to sequentially scan the tunnel lining surface at a stable camera angle at high speed, and it was shown that the acquired images could be stitched together to produce an expansion image that can be used for inspection.

In this study, we developed a real-time vibration visualization system that can estimate and display vibration distributions at all frequencies in real time through parallel implementation of subpixel digital image correlation (DIC) computations with short-time Fourier transforms on a GPU-based high-speed vision platform. To help operators intuitively monitor high-speed motion, we introduced a two-step framework of high-speed video processing to obtain vibration distributions at hundreds of hertz and video conversion processing for the visualization of vibration distribution at dozens of hertz. The proposed system can estimate the full-field vibration displacements of 1920 × 1080 images in real time at 1000 fps and display their frequency responses in the range of 0–500 Hz on a computer at dozens of frames per second by accelerating phase-only DICs for full-field displacement measurement and video conversion. The effectiveness of this system for real-time vibration monitoring and visualization was demonstrated by conducting experiments on objects vibrating at dozens or hundreds of hertz.

The acquisition of micro-vibrations is important for analyzing machinery. In the present study, we propose a method for measuring and visualizing the three-dimensional (3D) displacements of such micro-vibrations, especially in the case of sound waves propagating through space. The proposed method uses the speckle patterns of coherent light to measure the minute displacements. Speckle patterns are useful for detecting extremely small displacements owing to their sensitivity to the pose of the object. However, it is impossible to measure the displacement at each position because the pattern changes nonlinearly with respect to large depth changes. Therefore, a method of nonlinear low-dimensional embedding of the speckle pattern is proposed to analyze the displacements and extended to measure micro-displacements in a 3D space. We divided the 3D space into multiple slices and synchronously captured each speckle pattern. The displacements in the entire 3D space were simultaneously recovered by optimizing the embedded vectors, which were consistent in a 3D lattice. The propagation of sound waves in the 3D space was visualized using the volume-rendering technique. The experiments confirmed that the proposed method correctly measured the displacements by comparing them with the ground truth captured by microphones. We also visualized the wavefront of the sound wave propagating through space.

In this study, we propose a method for measuring the distance and velocity of overlapping vehicles using high-speed stereo vision. The method is based on the correlation filter called MOSSE and is improved by balancing the computation cost and accuracy using the features of high-frame-rate images. To achieve stable tracking, the tracking window was adjusted for each vehicle according to the vehicle distance and the overlapping state. Experiments on the images acquired by the cameras mounted on an actual vehicle were performed to validate the proposed method.

Multiple-target tracking across multiple camera views is required for various practical tracking and surveillance applications. This study proposes a seamless tracking method based on label transfer using high-speed image capture. The proposed method does not require camera calibration to estimate the extrinsic parameters, and can be applied with simple line calibration. Therefore, it is effective for large-scale camera networks. In addition, owing to the features of the high-speed image capture and self-window, our method ensures seamless tracking of objects across multiple camera views. The feasibility of the method is proven through an experiment using a high-speed camera network system.

In this study, we propose a multi-thread artificial intelligence (AI) camera system that can simultaneously recognize remote objects in desired multiple areas of interest (AOIs), which are distributed in a wide field of view (FOV) by using single image sensor. The proposed multi-thread AI camera consists of an ultrafast active vision system and a convolutional neural network (CNN)-based ultrafast object recognition system. The ultrafast active vision system can function as multiple virtual cameras with high spatial resolution by synchronizing exposure of a high-speed camera and movement of an ultrafast two-axis mirror device at hundreds of hertz, and the CNN-based ultrafast object recognition system simultaneously recognizes the acquired high-frame-rate images in real time. The desired AOIs for monitoring can be automatically determined after rapidly scanning pre-placed visual anchors in the wide FOV at hundreds of fps with object recognition. The effectiveness of the proposed multi-thread AI camera system was demonstrated by conducting several wide area monitoring experiments on quick response (QR) codes and persons in nature spacious scene such as meeting room, which was formerly too wide for a single still camera with wide angle lens to simultaneously acquire clear images.

Object tracking and pose estimation have always been challenging tasks in robotics, particularly for rotating objects. Rotating objects move quickly and with complex pose variations. In this study, we introduce a marker-based tracking and pose estimation method for rotating objects using a high-speed vision system. The method can obtain pose information at frequencies greater than 500 Hz, and can still estimate the pose when parts of the markers are lost during tracking. A robot catching experiment shows that the accuracy and frequency of this system are capable of high-speed tracking tasks.

In multi-object tracking of identical objects, it is difficult to return to tracking after occlusions occur due to three-dimensional intersections between objects because the objects cannot be distinguished by their appearances. In this paper, we propose a method for multi-object tracking of identical objects using multiple high-speed vision systems. By using high-speed vision, we take advantage of the fact that tracking information, such as the position of each object in each camera and the presence or absence of occlusion, can be obtained with high time density. Furthermore, we perform multi-object tracking of identical objects by efficiently performing occlusion handling using geometric constraints satisfied by multiple high-speed vision systems; these can be used by appropriately positioning them with respect to the moving region of the object. Through experiments using table-tennis balls as identical objects, this study shows that stable multi-object tracking can be performed in real time, even when frequent occlusions occur.

Camera-based wide-area self-posture estimation is an effective method to understand and learn about human motion, especially in sports. However, although rapid spatial tracking typically requires markers, prepositioned markers require extensive preparation in advance, and area projection markers exhibit problems in bright environments. In this study, we propose a system for spatial tracking and graphics display using vector-based laser projection embedded with M-sequence dashed line markers. The proposed approach is fast, wide-area, and can operate in bright environments. The system enables embedding and calibration of M-sequence codes in non-circular vector shapes, as well as rapid image processing recognition. We verified that the accuracy and speed of the proposed approach sufficed through static and dynamic tracking evaluations. We also demonstrate a practical application.

In recent years, it is required to acquire three-dimensional information at high speed in various fields. Previously, a structured light field (SLF) method for high-speed three dimensional measurement in 1 ms was proposed by our group. However, the SLF method has a drawback of worse depth estimation error by several tens millimeters. In this paper, a novel method to generate SLF with two projectors placed in parallel is proposed. This arrangement could produce bigger pattern change depending on the depth and made more precise estimation possible. The depth estimation experiments for precision evaluation and dynamic projection mapping experiment successfully demonstrated precise depth estimation with the error of several millimeters and high-speed estimation within 1 ms, though the measurement range was limited to approximately 100 mm.

A wide range of research areas have high expectations for the technology to measure 3D shapes, and to reconstruct the shape of a target object in detail from multiple data. In this study, we consider a high-speed shape measurement technology that realizes accurate measurements in dynamic scenes in which the target object is in motion or deforms, or where the measurement system itself is moving. We propose a measurement method that sacrifices neither measurement density nor accuracy while realizing high speed. Many conventional 3D shape measurement systems employ only depth information to reconstruct a shape, which makes it difficult to capture the irregularities of an object’s surface in detail. Meanwhile, methods that measure the surface normal to capture 3D shapes can reconstruct high-frequency components, although low-frequency components tend to include integration errors. Thus, depth information and surface normal information have a complementary relationship in 3D shape measurements. This study proposes a novel optical system that simultaneously measures the depth and normal information at high speed by waveband separation, and a method that reconstructs the high-density, high-accuracy 3D shape at high speed from the two obtained data types by block division. This paper describes the proposed optical system and reconstruction method, and it evaluates the computation time and the accuracy of reconstruction using an actual measurement system. The results confirm that the high-speed measurement was conducted at 400 fps with pixel-wise measurement density, and a measurement accuracy with an average error of 1.61 mm.

In this paper, we developed a concept of video monitoring system using a single mirror-drive pan-tilt mechanism. The system provides multiple zoomed videos with controllable viewing angle for each zoomed video and a wide-angle video. The system can be accessed by several users by connecting their personal computer (PC) to the server PC through the network. Every user is granted to change of their respected viewing angle of zoomed videos. The system is suitable for the remote observation deck for sight-seeing purpose. The system is composed of two high-speed cameras with wide-angle and zoomed lens, and a high-speed mirror-drive pan-tilt mechanism. The system implements a convoluted neural network-based (CNN-based) object detection to assist every user client identifying objects appearing on wide-angle and zoomed videos. We demonstrated that our proposed system is capable to provide wide-angle and zoomed videos with CNN-based object detection to four clients, where each client receives a 30 frames per second zoomed video.

It is important to miniaturize robot systems while maintaining advantages such as high responsiveness and functionality for human-machine interactions and for achieving integration with other robotic systems such as drones. In this research, we focused on the miniaturization of a high-speed visual feedback system, and developed a “portable saccade mirror,” which is a system that can realize active target tracking using 1000 Hz image capturing, processing, and feedback actuation with only 3 ms latency in a handheld device. By using a three-dimensionally-stacked vision chip, the proposed system achieved high speed, low latency, low power consumption and compact size, and therefore, can be considered as a good example of a miniaturized high-speed visual feedback system. In this study, we evaluated the performance of the proposed system in comparison with the conventional optical gaze controller, and demonstrated some applications, such as tracking field scope and panorama target scanning.

High-frame-rate visual feedback has been proven effective for projection mapping on quickly moving targets. However, the quantitative conditions of frame rates have not been elucidated. In particular, considering that the human dynamic visual acuity generally declines as the moving target velocity increases, it may be argued that the tracking accuracy required for projection mapping on moving targets can be lower than that required for fixed targets. To examine the above-mentioned conditions of frame rates, this study compares them based on two types of projection mapping systems having equivalent relative movements to each other: projection mapping from a fixed projector/camera system to a moving target and projection mapping from a moving projector/camera system to a fixed target. We examined the effects of the target movements by having stationary observers visually recognize and evaluate them. Using the weighted up-down method, we measured the frame time that provides a just noticeable difference (JND) to the projection mapping. We found that when the visual feedback is generated at a frame time of 2 ms, the frame time that provides a JND is 3.72 ms on average for fixed targets and 3.94 ms on average for moving targets: a 1%-level significant difference. Taking the individual differences and experimental errors into account, this is not a very large variation. This suggests that, in projection mapping to moving targets, we should realize a tracking accuracy as precise as that in projection mapping to fixed targets.

Fundus images contain extensive health information. However, patients hardly obtain their fundus images by themselves. Although glare-free infrared (IR) imaging enables easy acquisition of fundus images, it is monographic and challenging to process in real-time in response to high-speed and involuntary fixational eye movement and in vivo blurring. Therefore, we propose applying our IR color technology and providing clear fundus images by high-speed tracking of involuntary fixational eye movements and eliminating in vivo blurs by deconvolution. We tested whether the proposed camera system was applicable in medical practice and capable of medical examination. We verified the IR color fundus camera system could detect ophthalmological and lifestyle-related diseases.

In this study, we propose a high-speed and low-latency 3D fluorescence imaging method for robotic microscopes. The prototype system consists of a focus-tunable lens called a TAG lens, which operates at several hundred kHz, an image intensifier (I.I.) that enhances faint light such as fluorescence, and a high-speed vision system that can transfer acquired images to the host PC in 500 Hz. The proposed method can acquire images at arbitrary focal lengths at frame rates on the order of 1 kHz by synchronizing the focal-length fluctuation of the TAG lens and the exposure timing of the I.I., whose duration is a few hundred nanoseconds. The low-latency we aim for in this paper is on the order of a few milliseconds. A prototype system was developed to validate the proposed method. High-speed 3D tracking of the Brownian motion of a fluorescent bead of 0.5 μm diameter was demonstrated to verify the feedback performance of the proposed low-latency 3D fluorescence imaging method.

By combining an aerial display, user tracking system, and virtual reality (VR) space, we realized an aerial interface for providing both an immersive user experience and real-time communication with people outside the device. We developed a simple application for detecting user contact and changing the color of an object to measure the latency of our aerial interface system.

This paper reports some results of network connectivity control experiments using a multi-robot system which we developed. Although a lot of connectivity control algorithms for a multi-robot network are proposed, almost all of them are verified only on computer simulations or using experimental robots with centralized sensors and controllers. To execute experimental verifications of connectivity control algorithms on a distributed robotic system, we developed an experimental multi-robot system. Hardware installed on the robot and information flow from sensors to actuators are detailed. Some results of measurement experiments are shown to estimate accuracy to detect a neighbor position. Then, results of connectivity control experiments using the developed multi-robot system are discussed.

A visitor-following method that guides visitors as they move around was successfully developed without changing the structure of the waypoint navigation system. We previously developed a guidance robot, “EM-Ro,” to provide guidance services at the ECO35 Muffler Museum, and used the waypoint navigation system to implement a visitor-escort method along a predetermined route. With this visitor-following method, EM-Ro was able to follow a target visitor along visitor-derived waypoints, which were estimated using 2D LiDAR. Thus, the proposed navigation system for the guidance robot provides both visitor-escort and visitor-following guidance services. Using the same waypoint navigation system, it was possible to seamlessly switch between visitor-escort and visitor-following guidance. Switching between prepared or visitor-derived waypoints can make a visitor choose the preferred guidance method. Visitors can switch the guidance method anytime by providing EM-Ro requests from the remote controller. In addition, a guest redetecting method was developed when EM-Ro lost guests. The experimental results at the Muffler Museum showed that both visitor-escort and visitor-following driving by the EM-Ro were successfully demonstrated while guiding guests in the facility.