Field robotics has been undergoing rapid progress in recent years. It addresses a wide range of activities performed in outdoor environments, and its applications are being developed in areas where it was previously considered difficult to apply. This rapid progress is largely supported by AI-based improvements in computer vision systems with monocular cameras, stereo cameras, RGB-D cameras, LiDAR systems, and/or other sensors. Field robotics is impelled by an application-driven approach by its nature, and it contributes to the resolution of social problems and the creation of new innovations, including autonomous driving to reduce casualties, autonomous working machines/robots to resolve the problems of labor shortages or dangers, disaster-response robots to aid rescue parties, various kinds of aerial robots to do searches or make deliveries, underwater robots to perform search missions, etc.

In this special issue on “Field Robotics with Vision Systems,” we highlight sixteen interesting papers, including one review paper, fourteen research papers, and one development report. They cover various application areas, ranging from underwater to space environments, and they propose interesting integration methods or element technologies to use in outdoor environments where vision systems and robot systems have great difficulty performing robustly.

We thank all authors and reviewers, and we hope that this special issue contributes to future research and development in area of field robotics, which promises new innovations.

Field robotics is an area that is impelled by an application-driven approach by its nature. In this paper, I first review certain actual application areas of field robotics. Then, I discuss the current status of the application of field robotics in three common technologies: (1) mapping and path planning; (2) self-localization, recognition, and decision-making; and (3) dynamics and control. I then conclude by presenting future perspectives.

The melting of ice and changes in ocean currents in Antarctica must be investigated to understand global climate change. In this regard, the volume changes of sea ice and ice shelves, bathymetry, and ocean currents in the Antarctic Ocean must be measured in three dimensions. Therefore, the use of autonomous underwater vehicles (AUVs), which can directly observe under ice, is being considered. The authors developed an AUV named Mobility Oriented Nadir AntarctiC Adventurer (MONACA) to observe sea ice and the lower region of the ice shelf in the Antarctic Ocean. Herein, we describe MONACA and its basic autonomous navigation methods (altitude control, depth control, and waypoint tracking), as well as report the results of a sea experiment conducted in Shimoda Bay, Japan. During the 5-day sea trial, the MONACA successfully measured bathymetry by tracking 15 waypoints in sequence, switching the control criteria in the -axis direction between 3 m depth and 3 m altitude.

This report describes a numerical and experimental study of a posture control device based on a movable float for portable underwater robots. We numerically analyzed the static stability using a stability curve and allowable spatial range of a center-of-gravity shift caused by a payload shift or manipulator configuration. Further, we proposed a feedback controller based on direct pitch and roll signals to change and maintain robot posture. We tested the feedback control using a numerical simulator and conducted experiments in a water tank using two portable underwater robots to demonstrate the effectiveness of the movable float device and proposed controller. The results of the field experiments showed that the device and proposed controller can be employed for effective underwater operations of portable underwater robots.

Hydraulic excavators are one type of construction equipment used in various construction sites worldwide, and their usage and scale are diverse. Generally, the work efficiency of a hydraulic excavator largely depends on human operation skills. If we can comprehend the experienced operation skills and utilize them for manual control assist, semi-automatic or automatic remote control, it would improve its work efficiency and suppress personnel costs, reduce the operator’s workload, and improve his/her safety. In this study, we propose a methodology to design efficient machine trajectories based on mathematical models and numerical optimization, focusing on ground-level excavation as a dominant task. First, we express its excavation trajectory using four parameters and assume the models for the amount of excavated soil and the reaction force based on our previous experiments. Next, we combine these models with a geometrical model for the hydraulic excavating machine. We then assign the amount of soil to a performance index preferably to be maximized and the amount of work to a cost index preferably to be minimized, both in the form of functions of the trajectory parameters, resulting in an optimization problem that trades them off. In particular, we formulate (1) a multi-objective optimization problem maximizing a weighted linear combination of the amount of soil and the amount of work as an objective function, and (2) a single-objective optimization problem maximizing the amount of soil under a given upper bound on the amount of work, so that we can solve these optimization problems using the genetic algorithm (GA). Finally, we conclude this paper by suggesting our notice on design methodology and discussing how we should provide the optimization method as mentioned above to the users who operate hydraulic excavators.

The paper discusses a camera-based velocity estimation for unmanned ground vehicles in an agriculture scale. The proposed concept-based method does not require any geometric feature and focuses on a mapping between the captured images only. The paper provides three pilot experiments. First, we check an assumption of the proposed concept by a field experiment. Second, we check the verification by a set of numerical and laboratory experiments. Third, we check the verification by the field experiment. In the sense that the existence and sensitivity of a representation of the mapping are verified experimentally, the feasibility of the proposed concept is confirmed.

We propose a robotic forklift system for stacking multiple mesh pallets. The stacking of mesh pallets is an essential task for the shipping and storage of loads. However, stacking, the placement of pallet feet on pallet edges, is a complex problem owing to the small sizes of the feet and edges, leading to a complexity in the detection and the need for high accuracy in adjusting the pallets. To detect the pallets accurately, we utilize multiple RGB-D (RGB Depth) cameras that produce dense depth data under the limitations of the sensor position. However, the depth data contain noise. Hence, we implement a region growing-based algorithm to extract the pallet feet and edges without removing them. In addition, we design the control law based on path following control for the forklift to adjust the position and orientation of two pallets. To evaluate the performance of the proposed system, we conducted an experiment assuming a real task. The experimental results demonstrated that the proposed system can achieve a stacking operation with a real forklift and mesh pallets.

In recent years, the declining and aging population of farmers has become a serious problem. Smart agriculture has been promoted to solve these problems. It is a type of agriculture that utilizes robotics, and information and communication technology to promote labor saving, precision, and realization of high-quality production. In this research, we focused on robots that can harvest tomatoes. Tomatoes are delicate vegetables with a thin skin and a relatively large yield. During automatic harvesting of tomatoes, to ensure the operation of the harvesting arm, an input by image processing is crucial to determine the color of the tomatoes at the time of harvesting. Research on robot image processing technology is indispensable for accurate operation of the arm. In an environment where tomatoes are harvested, obstacles such as leaves, stems, and unripe tomatoes should be taken into consideration. Therefore, in this research, we propose a method of image processing to provide an appropriate route for the arm to ensure easy harvesting, considering the surrounding obstacles.

In this study, we developed a system for calculating the relative position and angle between a mobile robot and a marker using information such as the size of the marker of the internal camera of the mobile robot. Using this information, the mobile robot runs autonomously along the path given by the placement of the marker. In addition, we provide a control system that can follow a trajectory using information obtained by recognizing the mobile robot when reflected in an external camera using deep learning. The proposed method can easily achieve autonomous path travel control for mobile robots in environments where GPS cannot be received. The effectiveness of the proposed system is demonstrated under several actual experiments.

Unmanned mobile robots in rough terrains are a key technology for achieving smart agriculture and smart construction. The mobility performance of robots highly depends on the moisture content of soil, and past few studies have focused on terrain classification using moisture content. In this study, we demonstrate a convolutional neural network-based terrain classification method using RGB-infrared (IR) images. The method first classifies soil types and then categorizes the moisture content of the terrain. A three-step image preprocessing for RGB-IR images is also integrated into the method that is applicable to an actual environment. An experimental study of the terrain classification confirmed that the proposed method achieved an accuracy of more than 99% in classifying the soil type. Furthermore, the classification accuracy of the moisture content was approximately 69% for pumice and 100% for dark soil. The proposed method can be useful for different scenarios, such as small-scale agriculture with mobile robots, smart agriculture for monitoring the moisture content, and earthworks in small areas.

This paper describes the development of a robust object tracking system that combines detection methods based on image processing and machine learning for automatic construction machine tracking cameras at unmanned construction sites. In recent years, unmanned construction technology has been developed to prevent secondary disasters from harming workers in hazardous areas. There are surveillance cameras on disaster sites that monitor the environment and movements of construction machines. By watching footage from the surveillance cameras, machine operators can control the construction machines from a safe remote site. However, to control surveillance cameras to follow the target machines, camera operators are also required to work next to machine operators. To improve efficiency, an automatic tracking camera system for construction machines is required. We propose a robust and scalable object tracking system and robust object detection algorithm, and present an accurate and robust tracking system for construction machines by integrating these two methods. Our proposed image-processing algorithm is able to continue tracking for a longer period than previous methods, and the proposed object detection method using machine learning detects machines robustly by focusing on their component parts of the target objects. Evaluations in real-world field scenarios demonstrate that our methods are more accurate and robust than existing off-the-shelf object tracking algorithms while maintaining practical real-time processing performance.

Indoor navigation plays an essential role in agricultural robots that operate in greenhouses. One of the most effective methods for indoor navigation is the spread spectrum sound (SS-sound) system. In this system, the time of arrival (ToA) of the spread spectrum modulated sound is used for localization. However, there is a near-far problem. Transmitting the SS-sound from multiple anchors using time division multiple access (TDMA) is adequate to solve the near-far problem. However, localization is impossible because the ToA from multiple anchors cannot be simultaneously acquired. To solve this problem, a method for combining the SS-sound system with TDMA and an inertial navigation system is proposed in this study. The effectiveness of the proposed method was demonstrated through numerical simulations of a ground robot and experimentally using a crawler robot.

In this study, the teleoperation robot control on a mobile robot with 2D SLAM and object localization using LPWAN is proposed. The mobile robot is a technology gaining popularity due to flexibility and robustness in a variety of terrains. In search and rescue activities, the mobile robots can be used to perform some missions, assist and preserve human life. However, teleoperation control becomes a challenging problem for this implementation. The robust wireless communication not only allows the operator to stay away from dangerous area, but also increases the mobility of the mobile robot itself. Most of teleoperation mobile robots use Wi-Fi having high-bandwidth, yet short communication range. LoRa as LPWAN, on the other hand, has much longer range but low-bandwidth communication speed. Therefore, the combination of them complements each other’s weaknesses. The use of a two-LoRa configuration also enhances the teleoperation capabilities. All information from the mobile robot can be sent to the PC controller in relatively fast enough for real-time SLAM implementation. Furthermore, the mobile robot is also capable of real-time object detection, localization, and transmitting images. Another problem of LoRa communication is a timeout. We apply timeout recovery algorithms to handle this issue, resulting in more stable data. All data have been confirmed by real-time trials and the proposed method can approach the Wi-Fi performance with a low waiting time or delay.

There are many reports of secondary damage to crews during firefighting operations. One way to support and enhance their activities is to get robots to track them and carry supplies. In this paper, we propose a localization method for stairs that includes scene detection. The proposed method allows a robot to track a person across stairs. First, the scene detection autonomously detects that the person is climbing the stairs. Then, the linear model representing the first step of the staircase is combined with the person’s trajectory for localization. The method uses omnidirectional imaging and point clouds, and the localization and scene detection are available from any posture around the stairs. Finally, using the localization result, the robot automatically navigates to a posture where it can climb the stairs. Verification confirmed the accuracy and real-time capability of the method and demonstrated that the actual crawler robot autonomously chooses a posture that is ready for climbing.

This paper presents an attractive force estimation system and an automatic activation system for an electropermanent magnet (EPM) for an inspection UAV. Adsorption to infrastructures for inspection at a distance is extremely difficult to perform safely because the operator cannot detect the state of adsorption of the drone equipped with a magnetic adsorption device. Therefore, in this paper, we clarify the relationship between the magnetic flux density and attractive force of the EPM through experiments, and develop an estimation algorithm for the attractive force based on the results. An automatic activation system, using the induced voltage in the coil when the EPM approaches the magnetic substance, is developed and mounted on a quadrotor for a flight experiment along with the estimation system for the attractive force. The developed system is verified using flight and adsorption experiments on the quadrotor.

In this study, we propose a method for CanSat to recognize and guide a goal using deep learning image classification even 10 m away from the goal, and describe the results of demonstrative evaluation to confirm the effectiveness of the method. We applied deep learning image classification to goal recognition in CanSat for the first time at ARLISS 2019, and succeeded in guiding it almost all the way to the goal in all three races, winning the first place as overall winner. However, the conventional method has a drawback in that the goal recognition rate drops significantly when the CanSat is more than 6–7 m away from the goal, making it difficult to guide the CanSat to the goal when it moves away from the goal because of various factors. To enable goal recognition from a distance of 10 m from the goal, we investigated the number of horizontal regions of interest divisions and the method of vertical shifts during image recognition, and clarified the effective number of divisions and recognition rate using experiments. Although object detection is commonly used to detect the position of an object from an image by deep learning, we confirmed that the proposed method has a higher recognition rate at long distances and a shorter computation time than SSD MobileNet V1. In addition, we participated in the CanSat contest ACTS 2020 to evaluate the effectiveness of the proposed method and achieved the zero-distance goal in all three competitions, demonstrating its effectiveness by winning first place in the comeback category.

Owing to the revision of Japanese building law in 2008, the demand for wall inspections has been increasing. Currently, wall inspections are performed by workers using hammering devices; this involves dangerous work at high elevations. Therefore, we developed an inspection system using NOBORIN®, a hanging-type wall climbing robot. In this paper, we introduce the robot and its hammering inspection system, and propose a method for image mosaicking and localization using images captured from an equipped camera. The estimated values are used to correct the elevation motion(s) of the robot.

Most simultaneous localization and mapping (SLAM) systems assume that SLAM is conducted in a static environment. When SLAM is used in dynamic environments, the accuracy of each part of the SLAM system is adversely affected. We term this problem as dynamic SLAM. In this study, we propose solutions for three main problems in dynamic SLAM: camera tracking, three-dimensional map reconstruction, and loop closure detection. We propose to employ geometry-based method, deep learning-based method, and the combination of them for object segmentation. Using the information from segmentation to generate the mask, we filter the keypoints that lead to errors in visual odometry and features extracted by the CNN from dynamic areas to improve the performance of loop closure detection. Then, we validate our proposed loop closure detection method using the precision-recall curve and also confirm the framework’s performance using multiple datasets. The absolute trajectory error and relative pose error are used as metrics to evaluate the accuracy of the proposed SLAM framework in comparison with state-of-the-art methods. The findings of this study can potentially improve the robustness of SLAM technology in situations where mobile robots work together with humans, while the object-based point cloud byproduct has potential for other robotics tasks.

In this study, we propose a tele-operated mobile manipulator for conducting underfloor work, including both inspection and repair. We focus on a caulking operation, and develop a mobile manipulator for finding and repairing cracks in a foundation. When designing the mobile manipulator, we consider a typical narrow underfloor environment, and perform detailed calculations. As a user interface, we furnish an image-pointing system for enabling us to operate the manipulator simply by clicking on a hand camera image. Then, we evaluate its performance based on navigation experiments, trajectory tracking experiments, and overall evaluation experiments.

Kinect has been utilized as a cost-effective, easy-to-use motion capture sensor using the Kinect skeleton algorithm. However, a limited number of landmarks and inaccuracies in tracking the landmarks’ positions restrict Kinect’s capability. In order to increase the accuracy of motion capturing using Kinect, joint use of the Kinect skeleton algorithm and Kinect-based marker tracking was applied to track the 3D coordinates of multiple landmarks on human. The motion’s kinematic parameters were calculated using the landmarks’ positions by applying the joint constraints and inverse kinematics techniques. The accuracy of the proposed method and OptiTrack (NaturalPoint, Inc., USA) was evaluated in capturing the joint angles of a humanoid (as ground truth) in a walking test. In order to evaluate the accuracy of the proposed method in capturing the kinematic parameters of a human, lower body joint angles of five healthy subjects were extracted using a Kinect, and the results were compared to Perception Neuron (Noitom Ltd., China) and OptiTrack data during ten gait trials. The absolute agreement and consistency between each optical system and the robot data in the robot test and between each motion capture system and OptiTrack data in the human gait test were determined using intraclass correlations coefficients (ICC3). The reproducibility between systems was evaluated using Lin’s concordance correlation coefficient (CCC). The correlation coefficients with 95% confidence intervals (95%CI) were interpreted substantial for both OptiTrack and proposed method (ICC > 0.75 and CCC > 0.95) in humanoid test. The results of the human gait experiments demonstrated the advantage of the proposed method (ICC > 0.75 and RMSE = 1.1460°) over the Kinect skeleton model (ICC < 0.4 and RMSE = 6.5843°).

This paper presents a new path planning algorithm for an autonomous mobile robot. It is desired that the robot reaches its goal in a known or partially known environment (e.g., a warehouse or an urban environment) and avoids collisions with walls and other obstacles. To this end, a new, efficient, simple, and flexible path finder strategy for the robot is proposed in this paper. With the proposed strategy, the optimal path from the robot’s current position to the goal position is guaranteed. The environment is represented as a grid-based map, which is then divided into a predefined number of subfields to reduce the number of required computations. This leads to a reduction in the load on the controller and allows a real-time response. To evaluate the flexibility and efficiency of the proposed strategy, several tests were simulated with environments of different sizes and obstacle distributions. The experimental results demonstrate the reliability and efficiency of the proposed algorithm.