Real-time anomaly detection system for large volume network traffic

Abstract

In recent years, due to the growing complexity of cyberattacks such as DDoS, the detection of network anomalies using conventional rules like threshold-detection is becoming increasingly difficult to accomplish. Therefore, a machine learning based novel network anomaly detection methodology with enhanced accuracy has been proposed. Furthermore, Internet traffic is increasing year by year due to the rapid penetration of IoT devices into every facet of society. Consequently, the use of a model involving massive computations for a huge data stream poses the problem of diminished real-time accuracy. In this study, we investigate a high-accuracy real-time anomaly detection system by combining the strengths of distributed infrastructure and deep learning for enhanced real-time efficiency. Specifically, by using Spark Streaming which is a distributed processing framework, and distributing the processing load to the trained machine learning model, real-time anomaly detection has been performed for traffic flowing on the Internet.