AutoGuard: A Dual Intelligence Proactive Anomaly Detection at Application-Layer in 5G Networks

Abstract

Application-layer protocols are widely adopted for signaling in telecommunication networks such as the 5G networks. However, they can be subject to application-layer attacks that are hardly detected by existing traditional network-based security tools that often do not support telecommunication-specific applications. To address this issue, we propose in this work AutoGuard, a proactive anomaly detection solution that employs application-layer Performance Measurement (PM) counters to train two different Deep Learning (DL) techniques, namely, Long Short Term Memory (LSTM) networks and AutoEncoders (AEs). We leverage recent advancements in Machine Learning (ML) that show the advantages brought by combining multiple ML models to build a dual-intelligence approach allowing the proactive detection of application layer anomalies. Our proposed dual-intelligence solution promotes signaling workload forecasting and anomaly prediction as a proactive security control in 5G networks. As a proof of concept, we implement our approach for the proactive detection of Diameter-related signaling attacks on the Home Subscriber Server (HSS) core network function. To evaluate our solution, we conduct a set of experiments using data collected from a real 5G testbed. Our results show the effectiveness of our dual intelligence approach on proactively detecting signaling anomalies with a precision reaching 0.86.

9 Appendix

1.1 9.1 Effect of the Aggregation Time Window on the Forecasting

Fig. 8.

Effect of the time window on the predictive performance of the FM.

The objective of this set of experiments is to evaluate the effect of the aggregation time window that is used for the generation of the statistical multivariate time series. To this end, we fix the look back and predict forward parameters to the pair (3, 3). From our empirical analysis, we found that when the time window is small (between two and six), the FM performs well in the prediction of the recurrent observations but fails in predicting the rare events compared with the same model trained using time series generated with larger time window values. This is mainly due to the fact that the statistical features generated over large time windows provides a better characterization of the time dependencies over sequences of observations. Since in our solution we are interested in forecasting anomalies, which are considered as rare event, we focus on evaluating the predictive performance related to larger time windows (beyond six). As illustrated in Fig. 8, the value 12 provides the best predictive performance among the large time window values. As such, we set the time window size to 12 for the remaining sets of experiments.

1.2 9.2 Hyper-parameters Tuning for the Forecasting Model

The objective of this set of experiments is to study the impact of the learning rate and the dropout regularization technique on the FM’s performance. While the learning rate controls how rigorous the model’s learning should be, the dropout allows preventing neural network models from over-fitting [53]. Following common practices, we vary the learning rate 1.00E−04 to 1.00E−01 and the dropout within the set {0.0, 0.2, 0.4, 0.6}. As reported in Table 1a, the best predictive performance is reached when the learning rate is equal to 0.001 with a negligible increase in the training time compared to larger learning rate values (i.e., 0.1 and 0.01).

We remark a notable increase in the training time between 1.00E−4 and 1.00E−3 for a marginal decrease in the prediction error. Though, the performance gain (in terms of RMSE) is significant from 1.00E−1 to 1.00E−2 and 1.00E−3 while the increase in time for 1.00E−2 and 1.00E−3 is marginal, therefore, the learning rate 1.00E−3 seems to achieve the best trade-off training time/predictive performance.

As for the dropout regularization, we consider the input dropout, which is applied to the input layer and the recurrent dropout that is applied to the recurrent input signal on the LSTM nodes. As depicted in Table 1b, both the input and the recurrent dropout do not have a significant impact on the model performance, however a slight improvement (smaller error) is achieved when no dropout is considered. Based on those findings, we fix the learning rate to 0.001 and avoid using the dropout for the remaining sets of experiments.

Table 1. Evaluating the effect of learning rate and dropout on the FM.