2023 Volume 31 Pages 562-577
Penetration testing is among the most efficient techniques to improve network system defense and search for potential weaknesses. Applying penetration testing with reinforcement learning can enhance automation and accuracy and reduce dependence on human labor. However, this approach still encounters obstacles in intricate network systems, such as large ones, where compromising is challenging. The lack of modeling derived from a specific common cybersecurity knowledge base also complicates effective applications in practice. Therefore, based on MITRE ATT&CK knowledge, we propose a multilayer action representation to improve the performance, accuracy, and applicability of penetration testing on complex networks. The multilayer action representation's goal is to embody actions in penetration testing as n-dimensional vectors while faithfully capturing their characteristics and relationships. Therefore, it directly improves the performance of reinforcement learning agents in large and complicated network scenarios. For faster training, we also use an epsilon-Wolpertinger architecture. We conducted experiments on four difficulty levels with three network configurations and 119 system scenarios and compared our approach with four different reinforcement learning techniques. Our approach not only represents and models actions with high accuracy but also improves the ability of reinforcement learning agents in a variety of difficult levels of network systems.
