Recent technological advancements have expanded the range of applications and possibilities for industrial robots.

Therefore, this special issue focuses on a wide range of topics, from reports of successful practical applications to ambitious proposals that could have a significant impact on the field in the future. These topics concern the automation of manual work, such as material handling in factories, logistics, commercial facilities, and public facilities, as well as elemental technologies and system integration for new work methods, such as collaboration and remote operation.

Specifically, this issue contains 20 papers. These include studies on elemental technologies, including traditional technologies such as sensing, computer vision, grasping, manipulation, mechatronics, control methods, robot hands, mechanisms, and robot programming; application technologies for the automation of picking and assembling; and proposals for the future with challenging technologies such as human-machine interaction, virtual reality, cyber-physical systems, foundation models, deep learning, soft robotics, systems integration, humanoids, and new automation system applications.

The editors hope that all readers will enjoy the selected papers. They would also be pleased if they could expand this topic area and promote further contributions inspired by this issue, and thank the authors and reviewers for their dedicated time and effort.

Training deep-learning-based vision systems requires the manual annotation of a significant number of images. Such manual annotation is highly time-consuming and labor-intensive. Although previous studies attempted to eliminate the effort required for annotation, the effort required for image collection was retained. To address this issue, we propose a human-in-the-loop dataset-collection method using a web application. To counterbalance workload and performance by encouraging the collection of multi-view object image datasets enjoyably, thereby amplifying motivation, we propose three types of online visual feedback features to track the progress of the collection status. Our experiments thoroughly investigated the influence of each feature on the collection performance and quality of operation. These results indicate the feasibility of annotation and object detection.

The product arrangement robot, which displays products at retail stores, is one of the applications of industrial robot arms. For automatic product arrangement using robot, it is necessary to detect the products to be arranged. Because the products to be arranged can exist in countless states, it is difficult to define all states of products to be arranged. Therefore, in this study, we focus on the fact that the products to be arranged are in an anomaly state and consider the use of anomaly detection. However, there are two problems associated with product detection using anomaly detection. First, the anomaly area estimated using anomaly detection does not correspond to the product area, which is ambiguous. Second, the anomaly product area that can be detected depends on the sensors used for the anomaly detection. For the first problem, we utilize the segmentation foundation model “Segment Anything.” Using the coordinates calculated based on the anomaly area as a prompt, it is possible to accurately extract only the products to be arranged. For the second problem, we define a new third state, “indetermination” in addition to normal and anomaly. By selecting the anomaly detection neural network (NN) that is not in an indetermination from among multiple NN, products to be arranged can be correctly detected. The comparison results of the single anomaly detection NN and our proposed method showed that the detection accuracy of the product areas improved from 46.5% to 71.6%. Furthermore, a robot using the proposed method successfully picked the products to be arranged up from the shelves.

The development of product display robots in retail stores is progressing as an application of industrial robots. Pose estimation is necessary to enable robots to display products. PYNet is a high-accuracy pose estimation method for the poses of simple-shaped objects such as triangles, rectangles, and cylinders, which are common in retail stores. PYNet improves pose estimation accuracy by first estimating the object’s ground face. However, simple-shaped objects have symmetrical shapes, making it difficult to estimate poses that are inverted by 180°. To solve this problem, we focused on the upper part of the product, which often contains more information, such as logos. We developed a new method, PYNet-zmap, by enabling PYNet to recognize the height direction, defined as the direction from the bottom to the top of the product. Recognizing the height direction suppresses the 180° inversion in pose estimation and facilitates automatic product display. In pose estimation experiments using public 3D object data, PYNet-zmap achieved a correct rate of 79.2% within a 30° error margin, an improvement of 2.8 points compared with the conventional PYNet. We also implemented PYNet-zmap on a 6-axis robot arm and conducted product display experiments. As a result, 82.5% of the products were displayed correctly, an improvement of 8.9 points compared to using PYNet for pose estimation.

Labor shortages are becoming a significant issue in manufacturing sites, and automation of bin picking is required. To automate bin picking, it is necessary to estimate the pose of objects. Traditionally, the poses of multiple objects are estimated. However, estimating the poses of multiple objects is difficult, because it requires accurate pose estimation even for objects that are overlapped with others. This study is based on the fact that a robot can only grasp one object at a time. We developed a method that selects an easily graspable object first and focuses on pose estimation for this single object. In the Siléane dataset, the accuracy of pose estimation was 86.1%, an improvement of 17.5 points compared with the conventional method, PPR-Net.

Product display robots are considered for industrial arm robot applications. Object pose estimation is necessary to automate product displays. However, the shapes of some objects in retail stores are simple, and robots often use RGB images from a single viewpoint. Consequently, the pose estimation accuracy is low depending on the viewpoint. Therefore, this paper proposes a multiview pose estimation method that fuses features using weights for each viewpoint. To calculate the weights, we focus on a shared object representation that expresses object poses through classification. The classification score for each class increased when pose estimation became easier. Thus, we developed a method that weighs features from each viewpoint using classification scores as confidence, and estimates the object pose. We compared the pose estimation results with those of the conventional method, which derives the most plausible pose from multiple estimation results. When the permissible angle error was set to 30°, the success rate of our method was 68.0%, which was 8.2 points higher than that of the conventional method.

To cope with variable types and quantities of production in factories and a wide variety of orders in distribution centers, the demand for autonomous mobile robots (AMRs), which are not restricted by guide rails, is increasing in place of automated guided vehicles (AGVs). Accurate localization and orientation estimation are necessary to realize a stable and highly efficient AMR transport system. Even if we assume that the cooperative robots compensate for localization and orientation errors generated by AMRs, the errors generated by current AMRs are larger than the operating range of the cooperative robots, and it is necessary to ensure that the errors are at least within the range. To realize this, it is important to investigate how an AMR changes its localization and orientation accuracy while operating and to correct its position and orientation. In this study, we measured the actual pose of an AMR using a ceiling camera while the AMR was operating and examined the accuracy of the AMR’s localization and orientation sequentially when the course geometry changed. We then used these data to correct the position and orientation. It was found that a linear relationship was observed between the orientation estimation error and angular velocity, and the accuracy could be improved by modifying the orientation estimates using this relationship. The accuracy of localization and orientation estimation could be improved by training neural networks with sine and cosine data instead of angular data to learn errors from the ceiling camera and simultaneous localization and mapping (SLAM) data, and correcting errors.

Manipulating deformable linear objects (DLOs) presents significant challenges, particularly in industrial applications. However, most previous research has primarily focused on the manipulation of single DLOs, leaving the problem of picking up stacked DLOs without causing collisions largely unexplored. This task is common in industrial production processes, such as the sorting of pipes or cables. This paper introduces a novel strategy to address this issue using a simple setup with only a depth sensor. The proposed approach begins by acquiring a point cloud of the scene. The supervoxel method is then employed to segment the points into several clusters. Based on these clusters, a graph is constructed and pruned by graph theory to search for DLO axes. The axes belonging to the same DLO are subsequently connected, and occlusion relationships are identified. Finally, the best target, optimal picking point, and motion planning strategy are evaluated. Experimental results demonstrate that this method is highly adaptable to complex scenarios and can accommodate DLOs of varying diameters.

In this paper, we propose a method for selecting end-effectors based on depth images for a robot that performs picking tasks using multiple end-effectors. The proposed method evaluates the graspability of each end-effector in a scene by convolving a hand model, represented as a two-dimensional binary structure, with the depth image of the target scene. A key feature of the method is that it requires no pre-training and does not rely on object or environmental models, operating solely with simple models of the end-effectors. In picking experiments involving eight types of electronic components commonly used in factory automation, the proposed method effectively alternated between suction and two-finger grippers. Compared to other training-free end-effector selection methods and approaches using a single end-effector, the proposed method demonstrated an improvement of over 14% in grasp success rate compared to the second-best method.

To address labor shortages in retail stores, the automation of tasks, such as product display and disposal, using industrial robots is actively being promoted. These robots use suction grippers to grasp the products. However, conventional suction grippers often experience pad curling due to grasping pose errors, leading to air leaks and failed grasps. In this study, we proposed a curling prevention structure for suction pads to improve the grasping performance of suction grippers. Our solution focuses on a spring structure that reacts only to the parts where the force is applied. The proposed curling prevention structure consists of a ring that connects the cylinders with an elastic string. The cylinders passively applied force to the areas where curling occurred, thereby preventing curling. The design is highly versatile, as it is independent of the suction pad type. We performed experiments to evaluate the success or failure of grasping by varying the grasping angles of the robot arm for five types of products. The results showed that the range of grasping was expanded by up to 15° compared with that of conventional suction grippers. In addition, we performed experiments in which the robot grasped and conveyed products placed on store shelves using our gripper. The results showed improvements in both grasping and conveyance success rates.

Robot-based picking of diverse objects remains one of the critical challenges in the field of robotic automation research. In the automation of vision-based picking, methods based on large-scale pre-training or object shape models have been proposed. However, in many real industrial settings, it is often difficult to prepare the necessary pre-trained models or object shape models. One approach that does not require pre-training or object shape models is fast graspability evaluation (FGE). FGE efficiently detects grasp positions by convolving depth images with a cross-sectional model of the robot hand. However, a limitation of FGE is that it requires the grasp direction to align with the sensor’s line of sight, and it can only detect grasp positions with up to 4 degrees of freedom (DoF). In this paper, we extend FGE to 6DoF grasp position detection by calculating the 2 additional DoFs of grasp posture based on the normal directions around the object’s grasp region, obtained from the depth image. In picking experiments using three types of actual industrial parts, a comparison between the proposed method and FGE confirmed an average improvement of 10% in grasp success rates.

The development of large language models and vision-language models (VLMs) has resulted in the increasing use of robotic systems in various fields. However, the effective integration of these models into real-world robotic tasks is a key challenge. We developed a versatile robotic system called SuctionPrompt that utilizes prompting techniques of VLMs combined with 3D detections to perform product-picking tasks in diverse and dynamic environments. Our method highlights the importance of integrating 3D spatial information with adaptive action planning to enable robots to approach and manipulate objects in novel environments. In the validation experiments, the system accurately selected suction points 75.4%, and achieved a 65.0% success rate in picking common items. This study highlights the effectiveness of VLMs in robotic manipulation tasks, even with simple 3D processing.

A social issue owing to the aging of society in recent years is the daily removal of garbage by the elderly. This study proposes a gripper that can grip large, flexible, and heavy objects such as garbage bags as well as small and light objects such as plastic bottles, with the aim of automating the garbage-collection process. The proposed gripper consists of an opening/closing mechanism with a rhombic link structure and a contact area that makes contact with the object to be gripped. This mechanism is characterized by its power-saving effect as it converts the weight of the waste into gripping force through the rhombic structure. Therefore, minimizing power consumption is important for automating garbage collection with battery-driven mobile manipulators. This study describes the principle of operation of the rhombic link structure of the proposed gripper and compares it with conventional motor-torque gripping. Furthermore, by incorporating a flexible structure in the form of the gripping contact area, we verified that the proposed gripper is capable of gripping and operating a total of six types of garbage depending on the size (large or small), content (bottles or cans), and material of the garbage bags. The results show that the proposed gripper uses 60% less electricity than a parallel open/close type gripper and improves the success rate of gripping large garbage bags by 20% owing to the flexible structure of the contact area.

To apply dynamic compensation in increasing productivity and realizing high-speed manipulation and manufacturing, a wide-range 3D tracking system combining a high-speed 3D compensation module (also called a delta robot because of its structure) and a robotic arm is developed. During tracking, the cameras collect 1000 images per second, and the tracked object position in the real-world robotic-arm base coordinate system is calculated using the collected pixel data and camera calibration parameters. Corresponding commands are generated for the robotic arm and delta robot to maintain their positions relative to the tracked object. To generate these commands, encoder feedback from the robotic arm and delta robot is utilized. The cameras provide 1000-Hz visual feedback, and the high-speed compensation module provides 1000-Hz response to movement with high accuracy. The robotic arm achieves this accuracy over a wide range. Successful tracking along the xy-axis is achieved. Even when the object moves faster, the tracking accuracy is high along the x- and z-axes. This study demonstrates the feasibility of using a delta robot to achieve wide-range object tracking, which may contribute to higher-speed manipulation of robotic arms to increase productivity.

The number of logistics warehouses is increasing due to the expansion of the e-commerce market. Among warehouse tasks, the retrieval, transportation, and storage operations of containers holding products on shelves are becoming increasingly automated because of labor shortages and the physical strain of workers. However, automated warehouse systems face challenges, such as the requirement for extensive space for implementation, high maintenance costs owing to rail-based positioning, and high pre-adjustment costs due to positioning with augmented reality markers or other signs attached to each container and shelf. To solve these problems, we aim to develop a system that enables the retrieval, transportation, and storage operations of containers without relying on positioning tools such as rails or signs. Specifically, we propose a system that uses a fork-inspired hand to handle a drawer equipped with fork pockets for retrieval and storage operations while considering the benefits of using drawers in terms of space efficiency and product management. For positioning, we use visual servoing with images obtained from a camera mounted at the end of the robot arm. In this study, we implement the following three methods in the proposed system and conduct comparative studies: image-based visual servoing (IBVS), which uses luminance values directly as features; active visual servoing (AVS), which projects patterned light onto an object for positioning; and convolutional neural networks-based visual servoing (CNN-VS), which infers the relative pose by using the difference between the feature maps of desired and current images. The experimental results show that CNN-VS outperforms the other methods in terms of the operation success rate.

Robot vision (2D/3D) is one of the most important components for robotic arm applications, especially for accurate and intelligent manipulation in unstructured working environments. In this study, we focused on the great potential of high-speed 3D vision with frame rates exceeding 500 fps for commercial robotic arm applications. By utilizing a UR10e robotic arm with a newly developed high-speed 3D vision system that is capable of achieving 1,000 fps 3D sensing, we investigated the impact of feedback rate on tracking performance towards a dynamic moving target. The integration of the high-speed 3D vision for robotic arm applications was implemented under the direct and the dynamic compensation approaches. Comparative studies demonstrated that higher feedback rates improve tracking accuracy up to a particular saturation feedback rate, with the dynamic compensation approach reducing tracking error by 70% compared with direct integration.

This study aims to realize a precision assembly robot system. This system focuses on the mating of precision parts, which is difficult with the current technology, and uses workpiece movements and the sensation of human fingertips as a reference to eliminate assembly failures, such as shaft and hole biting. This approach solves the limitations in robotizable tasks, long teaching times, and possible assembly failures by robots. The proposed method involves measuring human task data, analyzing the movement of assembly parts using averaging (a simple feature extraction method), and deriving the corresponding robot movements. Furthermore, a system is developed to predict decisive assembly failures from force information obtained during tasks by analyzing the force sensation of the human fingertips using a support vector machine (a type of machine learning). Equipped with the prediction system and the derived workpiece motion, the robot performs assembly tasks that typically require human skills.

The industrial sector has demonstrated increased interest in implementing in-hand manipulation as an alternative to conventional parallel gripper hands, enabling dexterous object handling with substantial pose changes. In-hand manipulation involves challenges such as complex contact transitions with robot fingers and difficult-to-model factors such as friction and surface irregularities, which complicate real-world applications despite successful simulations. This study proposes a novel approach for in-hand manipulation that outperforms traditional model-based motion-generation methods. A motion teaching system utilizing a leader–follower system was developed to address the limitations of conventional methods. This technique leverages human skills for robot motion teaching without requiring motion programming or precise modeling. The control system ensures stability during motion teaching, adapts to objects and operators with unknown dynamics, and can easily be extended to a motion-reproduction system. For motion-trajectory generation, we introduced a method that identifies highly reproducible motion trajectories by analyzing the similarity of time-series data from multiple teaching datasets. By selecting from the teaching data, we determine a motion trajectory that maintains a force consistent with the motion over time. Experiments validated the efficacy of the proposed system.

This study presents an error recovery architecture for future variable-mix variable-volume production based on cyber-physical-human systems (CPHS). It focuses on bin picking, which is a crucial manufacturing process for handling bulk industrial parts during kitting. One of the main challenges in bin picking is efficiently introducing perturbations to arbitrarily placed parts and make all parts graspable and resolve deadlock situations. For example, a suction-type gripper is advantageous for handling objects stably without geometric models as it can easily adhere to flat surfaces. However, the success rate of bin picking using a suction gripper depends on the orientation of the target part. If its flat graspable surface does not face upward, the suction gripper cannot attach to and pick up the target object, resulting in a deadlock. In this case, an external force must be applied to change the orientation of the target object to resume the bin-picking process. A conventional, albeit inefficient, solution is a human worker or an additional mechanism that perturbs the container. Because applying such a perturbation by a versatile robot is challenging due to the limited physical information, a promising approach for efficient error recovery is a combination of human remote instruction and automated trajectory planning. This study developed a CPHS-based architecture to facilitate error recovery through smooth human-robot collaboration. We perform three experiments to demonstrate the feasibility of this approach for efficient error recovery.

This paper describes a gesture interface for the operation of autonomous mobile robots (AMRs) for transportation in industrial factories. The proposed gesture interface recognizes pointing directions by human operators, who are workers in the factory, based on deep learning using images captured by a fovea-lens camera. The interface could classify pointing gestures into seven directions with a recognition accuracy of 0.89. This paper also introduces the navigation method for AMR to implement the proposed interface. This navigation method enabled the AMR to approach the pointed target by adjusting its horizontal angle based on the object recognition using RGB images. The AMR achieved high position accuracy with a mean position error of 0.052 m by implementing the proposed gesture interface and the navigation method.

This paper presents a virtual reality and digital twin-based training system designed to improve human-robot collaboration in retail store environments, particularly under disaster scenarios. This system enables dynamic role adaptation between humans and AI-controlled avatars or robots, facilitating diverse collaborative configurations. In a virtual retail environment replicating post-disaster conditions, human subjects—paired either with AI or another human participant—engage in collaborative object-retrieval tasks, distinguishing between safe and hazardous items. Experimental results indicate that human-human collaborations outperform human-AI collaborations in both task efficiency and safety. Participants exhibited improved movement efficiency and higher accuracy in retrieving safe items when paired with another human. These findings suggest that human-robot interaction training can benefit from human-human collaboration configurations for skill enhancement. This system also demonstrates potential for broader applications in simulating complex hazardous environments where real-world training is challenging.

This paper proposes a mobile quad-arm robot, ARMS, that unifies wheeled-legged tripedal locomotion, wheeled locomotion, and loco-manipulation. ARMS’s four arms have different mechanical configurations for hybrid locomotion and loco-manipulation and are partially designed to be general-purpose arms. The one three-degree-of-freedom (DOF) arm has an active wheel that is used for wheeled-legged tripedal walking and wheeled driving with passive wheels attached to the torso. The two three-DOF general-purpose arms are series elastic and are used for wheeled-legged tripedal walking, object grasping, and manipulation. The upper two-DOF arm is used for manipulation only, and its position and orientation are determined by coordinating all the arms. Each motor is controlled using an angle controller and trajectory modification with angle, angular velocity, angular acceleration, and torque constraints. The capabilities of ARMS were verified with seven experiments involving joint control, wheeled-legged locomotion, wheeled locomotion and grasping, slope locomotion, block terrain locomotion, carrying a bag, and outdoor locomotion.

In this paper, we propose a method for manipulating home appliances using arm-pointing gestures. Conventional gesture-based methods are limited to home appliances with known locations or are device specific. In the proposed method, the locations of home appliances and users can change freely. Our method uses object- and keypoint-detection algorithms to obtain the positions of the appliance and operator in real time. Pointing gestures are used to operate the device. In addition, we propose a start gesture algorithm to make the system robust against accidental gestures. We experimentally demonstrated that using the proposed method, home appliances can be operated with high accuracy and robustness, regardless of their location or the user’s location in real environments.

The operational ease of fixed-wing vertical take-off landing (VTOL) unmanned aerial vehicle (UAVs) derives from their capability to take off and land vertically. This particular capability is achieved through the utilization of an additional rotor propulsion system that operates during take-off and landing. However, the rotor propulsion system, located externally to the airframe, contributes to increasing drag force, especially during the cruise phase, reducing the efficiency and flight time. To overcome the issues, this study proposes a design for a fixed-wing VTOL UAV with the four-retractable rotor propulsion, demonstrating its feasibility and performance through flow simulation and flight tests. During flight tests at a speed of 18 m/s, UAVs with (folded UAV) and without retractable propulsion systems (unfolded UAV) can maintain cruise speeds of 18 m/s at throttle openings of 56.3% and 72.1%, respectively. The energy consumption was reduced by 33.0% followed by increasing in endurance of 32.0%. It aligns with the simulation results, which show that the four-retractable rotor propulsion can reduce the drag coefficient by 35.9% and increase the aerodynamic efficiency (C_L/C_D) by 58.3% compared to the unfolded UAV. Both results confirm that the four-retractable rotor propulsion significantly reduces the aerodynamic drag, and increases the efficiency and endurance.

This study proposed a fully automatic electric wheelchair control system based on an obstacle-shape measurement algorithm using a monocular camera and laser. When the electric wheelchair, equipped with the camera and laser, encounters an obstacle, the control system automatically avoids it in real time by measuring the distance and depth to the obstacle. As this research focuses on obstacle avoidance, a simple pathfinding algorithm is used. When an obstacle is detected, the system creates an avoidance path corresponding to the depth length of the obstacle, updates the current position, and moves to the goal point. The novelty of this system is that it calculates not only the distance to the obstacle but also the depth of the obstacle, which is three-dimensional information. Meanwhile, the minimum and maximum distances from the camera to the obstacle are detected by laser. The distance formula for the proposed method is not the straight-line distance from the camera, but the distance from the position in which the camera is moved horizontally parallel to the obstacle in front of the camera. Therefore, the obstacle depth can be calculated by the difference between the maximum and minimum distances from the camera to the obstacle. The effectiveness of the proposed method was verified by preparing obstacles with different depths and verifying whether the electric wheelchair could generate an avoidance route for each obstacle in the route search and still move to the goal point. We confirmed that the wheelchair stops when the distance to the obstacle reaches an arbitrary length, acquires information on the obstacle depth, creates an avoidance route according to the depth, avoids the obstacle, updates its current position, and automatically moves to the goal point. By further improving the effectiveness of the proposed method, a fully automatic electric wheelchair that can detect the shape of obstacles and avoid them appropriately in hospitals and on roads without depending on the illumination of the measurement environment can be realized.

The convenience store industry is experiencing a growing labor shortage, and the need to automate tasks is increasing. Product display is a labor-intensive task, and product recognition is an important issue. Existing recognition methods using deep learning require relearning every time a new product is introduced, which is time-consuming. In this study, a packaging design was developed that streamlines the learning process by embedding prelearned patterns and markers into the product packaging. The proposed design consists of patterns for product identification and markers for estimating product position and orientation. These are “typographic patterns” that change the characters and their minimum unit composition, as well as the manner in which the minimum units are arranged among themselves, and can create more than 400,000 different types of any products. This paper describes the creation of the proposed patterns and marks. The proposed design was then applied to a sandwich package, and identification experiments were conducted for 23 basic placement patterns. The identification rate was over 97%.