Abstract
The IP Multimedia Subsystem (IMS) has been constantly evolving to meet the tremendous rise in popularity of mobile services and Internet applications. Since IMS uses Session Initiation Protocol as the main protocol to control a signal, it inherits numerous known security vulnerabilities. One of the most severe issues is the Denial of Service attack. To address this problem, we introduce an anomaly-based detection system using the Tanimoto distance to identify deviations in the traffic. A modified moving average is applied to compute an adaptive threshold. To overcome a drawback of the adaptive threshold method, we present a momentum oscillation indicator to detect a gradually increasing attack. Generally, anomaly-based detection systems trigger many alarms and most of them are false positives that impact the quality of the detection. Therefore, we first present a false positive reduction method by using a trust model. A reliable trust value is calculated through the call activities and the human behavior of each user. The system performance is evaluated by using a comprehensive synthetic dataset containing various malicious traffic patterns. The experimental results show that this system accurately identified attacks and has the flexibility to deal with many types of attack patterns with a low false alarm.