The design of a web-based master system for robot control with ROS environment
2023 Volume 4 Issue 2 Pages 60-65
Details
2023 Volume 4 Issue 2 Pages 60-65
Despite recent advancements in autonomous construction using robotics technology (RT), challenges persist in terms of limited user accessibility, interoperability across different robot platforms, and the absence of effective communication mechanisms for intuitive human-robot interaction (HRI). This study addresses these challenges by leveraging web technologies and visual programming to develop a master system that combines intuitive robot control program prototyping and interaction. The web interface utilizes JavaScript libraries such as roslibjs, ros2djs, and ros3djs to enable intuitive robot control and visualization of camera views and sensor data. It acts as a remote hub for centralized control and management of robot fleets from a unified internet-based location, enhancing coordination and operational efficiency. Furthermore, the rapid prototyping process utilizes Matlab Simulink for algorithm development and code generation, with performance simulation and assessment in Unity. Validated control nodes are then transferred to real machines via cloud services, enabling accelerated development of efficient and robust robotic systems.
The traditional construction industry has been known as a labor-intensive industry where most construction activities rely heavily on human labor and manual operations1). However, an aging workforce and difficulties in attracting younger workers have resulted in a severe labor shortage. This calls for urgent work innovation to address the declining workforce. Recent advancements in information and communication technology (ICT)2) and robotics technology (RT) have opened the door for such transformative changes in the industry3). Particularly, the emergence of automated and autonomous construction machinery makes it possible to perform repetitive and labor-intensive tasks with minimal human involvement4). This frees up human workers to concentrate on more complex and specialized assignments, ultimately boosting overall productivity.
Construction machinery manufacturers and general contractors have made significant progress in developing their own autonomous robots and dedicated control systems5) 6) 7) 8). However, challenges remain in terms of limited user accessibility and interoperability across various robot platforms due to restricted user interaction within their respective control systems. Therefore, the need arises for an effective communication mechanism for quick and intuitive human-robot interaction (HRI), enhancing overall usability and collaboration among robots from different manufacturers. Recent advancements in web technologies have propelled the application fields of RT into new frontiers. Accordingly, this study proposes a standardized framework that seamlessly integrates robots with web technologies, enabling developers to interact intuitively and responsively with robots through web browsers.
The construction industry needs to sustain its production capacity to meet future construction project demands despite the challenges posed by an aging population and labor shortage9). Autonomous construction, with its potential for labor savings and improved working conditions, is viewed as a crucial technology for future construction sites. The recent research advancements have provided promising prospect for leveraging RT in construction automation, whereas the focus has primarily been on the autonomous operation of individual robots10). A few construction companies took a further step by integrating multiple construction robots for coordinated work for earthwork projects7) 8), whereas the scope of application remains limited. Noticeably, most of these construction robots are proprietary and developed with their own control systems, which hinders interoperability and coordination among robots from different manufacturers. To address this issue, the Public Works Research Institute (PWRI) of Japan introduced the concept of a common control signal that standardizes data exchange format between construction equipment and user-side systems. The concept is part of the boarder OPERA (Open Platform for Earthwork with Robotics and Autonomy) initiative, which aims to provide an open platform for faster technological development in autonomous construction11).
On the other hand, recent web technology development has brought significant advancements in cloud robotics2) 12). Cloud-enabled robotics system leverages the power of cloud computing resources to ensure scalability of robotics application and offer a remote and collaborative platform for developers, aligning with the concept of an open platform. Nevertheless, current research predominantly focuses on expanding off-board computational power13) 14) without regard to broader interoperability and human accessibility. To address this, a userfriendly and accessible control interface is crucial for robust HRI. In addition, developing a control program for autonomous coordinated operation of construction machines on-site can be challenging and time-consuming due to complex coding requirements. The ever-changing nature of the construction environment necessitates continuous adaptation of machinery behavior. There hence arises a need for a rapid prototyping mechanism that allows for quick development, validation, and deployment of new operational behavior logics tailored to sitespecific requirements.
The objective of this study is therefore divided into two parts. Firstly, to develop an easy-accessible and intuitive user interface (UI) that ensures smooth HRI, and a web-based solution is intended for this purpose. Secondly, to explore a rapid prototyping mechanism for quickly iterating on new operational behavior logics of robotic programs in accordance with site demands.
(1) System requirements
In pursuit of a more intuitive and effective way of prototyping and interacting with robot control programs, this study aims to design a master system with the following requirements -
a) Robot operation control
A user-friendly UI for basic robot operation control is needed. The UI should incorporate interactive elements, such as sliders and buttons, to control robot movements, actions, and parameters. Robot control programs should also be implementable from the UI.
b) Visual feedback
Visual feedbacks such as camera views and sensor data from a robot should be included in the UI for enhanced situational awareness and ensuring users to make informed decisions and adjustment promptly.
c) Web-based interface
Adopting a web-based approach for robot control interface eliminates geographical limits and the need for any special software installation, making it easier for users to engage with the robot control UI using their preferred devices at preferred locations. It also enables centralized control and management of the robot fleet from a single, unified web interface, allowing users to handle multiple robots simultaneously and coordinate their operations efficiently.
d) Communication protocols
A suitable communication protocol is required to establish effective communication channel between the UI and robots in both simulation and real environment. The protocol should offer a bidirectional and low-latency communication to ensure seamless control experience.
e) Visual programming-based development environment
Achieving autonomous coordinated action among construction machines is challenging due to complex coding requirements, while the dynamic nature of the construction environment calling for constant adjustment of machinery behavior further pushes the coding hurdles. To address this, a rapid prototyping mechanism is needed to iterate on new operational behavior logics as per site’s needs. Visual programming tools enable prompt development and testing of these logics through an intuitive drag-and-drop interface, facilitating the development of highlevel robot control algorithms without complex coding requirements.
(2) Generic overview of a proposed master system configuration
The master system with requirements described above is designed to have such architecture that ensures intuitive HRI and algorithm prototyping. Fig.1 gives a generic overview of the proposed webbased master system for robot control. Firstly, a visual programming-based environment translates the site’s demand for specific coordinated operation of construction machinery into corresponding logic flows. A web-based robot control application then executes the program built from the logic flows and passes down the control signals to a simulator where the developed logics are evaluated. The simulator is capable of taking data of various format to configure a high-fidelity simulation environment for precise simulation of machine behavior. Any issues identified during the simulation step will be fed back for revision and re-assessment. Upon simulation validation, the control algorithms are built into executable robot control programs that may be deployed onto real machines. This iterative process allows for continuous improvement of the control algorithms, ensuring optimized performance of coordinated operation of construction machinery in a real-world setting.
The aforementioned generic system configuration has been devised with the objective of establishing a standardized browser-based robot control framework for intuitive robot control and rapid prototyping of control programs. Fig.2 illustrates the overall flow of the master system. In order to fulfill the proposed master system a range of tools and software have been employed.
(1) Robot control Web application
The Robot Operating System (ROS), an opensource robot middleware which provides extensive libraries and tools for robot application development, is used as a backend server. To enable communication between external clients, such as web browsers and ROS devices, an established communication protocol is necessary and rosbridge15) is adopted in this system. rosbridge offers a JSON interface for non-ROS clients to interact with ROS functionality. This integration is part of the Robot Web Tool (RWT) project16), an open-source project facilitating communication between robots and web-based interfaces. Through rosbridge, the client library roslibjs17) is then able to interact with ROS topics, services, and actions from the web UI it was embedded. Two additional JavaScript libraries from RWT, namely ros2djs18) and ros3djs19), were employed to help visualize 2D and 3D ROS data types such as maps, URDFs, laser scans, and point clouds. Both libraries utilize roslibjs and rendering libraries such as EaseIJS20) and three.js21) to create interactive scenes on the HTML5 <canvas> element. Additionally, the web-based interface supports images and camera streams through the web_video_server ROS package22). These visual feedbacks help to enhance situational awareness, allowing users to make informed decisions and timely adjustments.
An example of robot control UI utilizing the above JavaScript libraries is shown in Fig.3, which contains an input field for specifying the rosbridge WebSocket server IP address to establish a connection with external ROS devices, a panel for sending movement commands, calling services, and executing actions, and a canvas for streaming camera views captured by a robot.
(2) Visual programming & logic implementation
Matlab Simulink®, with its seamless integration with ROS via the dedicated ROS Toolbox, serves as the visual programming tool for rapid development of robot control algorithms. The Simulink Coder Toolbox® generates the ROS package source code based on the logic flow models and builds the standalone ROS node executables on a remote ROS device. In Simulink it is possible to configure ROS network address to ensure connection to the target ROS device. The Simulink-generated ROS node is then executed by the ROS device, and the outcomes are shown in a simulated environment, as is demonstrated in Fig.4.
(3) Simulator
In order to validate the control logics for coordinated operation among multiple robots, a simulation environment that depicts real-world construction conditions is required. Unity®, a widely used game engine, provides robust rendering capabilities and the ability to configure high-fidelity virtual environment with extensive data exchange framework23). Its low-latency communication with ROS via the Unity official ROS-TCP-Connector package24) also makes it an ideal choice as a simulator for the proposed master system. Furthermore, Unity’s potential integration with Machine Learning and Artificial Intelligence (AI) facilitates the training and assessment of AI algorithms within the simulation environment, helping to achieve the ultimate objective of autonomous construction.
(4) Explanation on system overview
As is demonstrated in Fig.2, the top component is a robot control web application developed using HTML and roslibjs. This web application acts as a hub for centralized control and management of robot fleets. Facilitated by ros2djs and ros3djs, a customizable dashboard is also included in the UI for visualizing sensor data and camera images capture by robots. Client devices, such as laptops or tablets, can access the web application remotely via HTTP request to control and monitor robots. The web application uses the rosbridge protocol to communicate with a remote ROS device, which is any Linux system that has ROS installed. This communication via rosbridge allows web users to interact with the robot by publishing and subscribing to ROS topics, calling services, executing actions, and manipulating parameters without the need for a ROS installed platform. The ROS environments are divided into two parts, one for algorithm development and testing in a simulation space, and another for deploying verified control programs on real machines. Within the development environment, Matlab Simulink serves as a visual programming tool for rapid algorithm development and code generation. Control logics are initially developed as Simulink models and then built into standalone ROS node executables to be run on a remote ROS device A. The respective control performance of each ROS node is examined in a Unity-powered simulation environment through ROS-TCP-Connector package. Upon performance verification, the ROS nodes are transferred to another ROS device B responsible for real machine control. Node transfer can be accomplished using cloud services such as Github®. By cloning and running the node executables from the cloud, the registered real machines operate according to the defined logic in Simulink.
Despite recent research advancements in the field of RT for autonomous construction, challenges persist in terms of limited user accessibility, interoperability across different robot platforms, and the absence of an effective communication mechanism for intuitive human-robot interaction. Moreover, the dynamic nature of the construction environment requires constant adjustment of machinery behavior, necessitating a rapid prototyping mechanism for quick iteration on new operational behavior logics of machines. Accordingly, this study aims to leverage the advantages of web technologies and visual programming to develop a master system that combines intuitive prototyping and interaction with robot control programs. The web interface, responsible for intuitive robot control, utilizes JavaScript libraries like roslibjs to communicate with ROS backend servers via the rosbridge protocol. It also employs ros2djs and ros3djs to visualize camera views and sensor data, providing visual feedback for improved situational awareness. This approach enables users to make informed decisions and timely adjustments. By introducing such robotic interfaces, a broader user base can be reached without requiring software installation or set-up. Furthermore, the web interface enables centralized control and management of robot fleets from a unified internetbased location, facilitating efficient coordination of multiple robots and their operations.
Matlab Simulink software is employed as a visual programming tool for rapid algorithm development and code generation. Control logics are created as Simulink models, built, and executed as ROS nodes on a remote ROS device. The performance is simulated and analyzed in Unity, a simulator known for its strong physics engine and versatile data exchange framework to precisely simulate machine behavior in a realistic virtual environment. Subsequently, the validated nodes are transferred via cloud services like Github to another ROS device for real machine control. This rapid prototyping process facilitates accelerated development of robot control programs, and thereby a more efficient and robust robotic system.
Regarding the future work, integration with external construction management systems is crucial in enabling construction machinery to operate in line with specified construction plans. The master system should be able to parse and interpret construction tasks into machine movement and operational commands and thus to generate corresponding control programs. Furthermore, an interactive dashboard should be introduced to the web UI to monitor various IoT data retrieved from construction machines, giving insights to real-time machine performance, location, motion history, fuel consumption, and other metrics. To safeguard data transmission and prevent unauthorized access, robust cybersecurity measures including data encryption and user authentication should be implemented. Lastly, the web application should incorporate safety mechanisms like collision detection, avoidance algorithms, and emergency stop functionality to ensure safe machine operation.
