{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,8]],"date-time":"2024-09-08T14:59:38Z","timestamp":1725807578175},"reference-count":45,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,3,20]],"date-time":"2022-03-20T00:00:00Z","timestamp":1647734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"Network security encloses a wide set of technologies dealing with intrusions detection. Despite the massive adoption of signature-based network intrusion detection systems (IDSs), they fail in detecting zero-day attacks and previously unseen vulnerabilities exploits. Behaviour-based network IDSs have been seen as a way to overcome signature-based IDS flaws, namely through the implementation of machine-learning-based methods, to tolerate new forms of normal network behaviour, and to identify yet unknown malicious activities. A wide set of machine learning methods has been applied to implement behaviour-based IDSs with promising results on detecting new forms of intrusions and attacks. Innovative machine learning techniques have emerged, namely deep-learning-based techniques, to process unstructured data, speed up the classification process, and improve the overall performance obtained by behaviour-based network intrusion detection systems. The use of realistic datasets of normal and malicious networking activities is crucial to benchmark machine learning models, as they should represent real-world networking scenarios and be based on realistic computers network activity. This paper aims to evaluate CSE-CIC-IDS2018 dataset and benchmark a set of deep-learning-based methods, namely convolutional neural networks (CNN) and long short-term memory (LSTM). Autoencoder and principal component analysis (PCA) methods were also applied to evaluate features reduction in the original dataset and its implications in the overall detection performance. The results revealed the appropriateness of using the CSE-CIC-IDS2018 dataset to benchmark supervised deep learning models. It was also possible to evaluate the robustness of using CNN and LSTM methods to detect unseen normal activity and variations of previously trained attacks. The results reveal that feature reduction methods decreased the processing time without loss of accuracy in the overall detection performance.<\/jats:p>","DOI":"10.3390\/informatics9010029","type":"journal-article","created":{"date-parts":[[2022,3,21]],"date-time":"2022-03-21T01:30:14Z","timestamp":1647826214000},"page":"29","source":"Crossref","is-referenced-by-count":9,"title":["Benchmarking Deep Learning Methods for Behaviour-Based Network Intrusion Detection"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"http:\/\/orcid.org\/0000-0003-3448-6726","authenticated-orcid":false,"given":"M\u00e1rio","family":"Antunes","sequence":"first","affiliation":[{"name":"Computer Science and Communication Research Centre (CIIC), School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal"},{"name":"Institute for Systems and Computer Engineering, Technology and Science (INESC TEC, CRACS), 4200-465 Porto, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-9412-5012","authenticated-orcid":false,"given":"Lu\u00eds","family":"Oliveira","sequence":"additional","affiliation":[{"name":"Smart Cities Research Center (Ci2), Polytechnic Institute of Tomar, 2300-313 Tomar, Portugal"}]},{"given":"Afonso","family":"Seguro","sequence":"additional","affiliation":[{"name":"Polytechnic Institute of Tomar, 2300-313 Tomar, Portugal"}]},{"given":"Jo\u00e3o","family":"Ver\u00edssimo","sequence":"additional","affiliation":[{"name":"Polytechnic Institute of Tomar, 2300-313 Tomar, Portugal"}]},{"given":"Ruben","family":"Salgado","sequence":"additional","affiliation":[{"name":"Polytechnic Institute of Tomar, 2300-313 Tomar, Portugal"}]},{"given":"Tiago","family":"Murteira","sequence":"additional","affiliation":[{"name":"Polytechnic Institute of Tomar, 2300-313 Tomar, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1080\/23742917.2018.1518061","article-title":"KDD 1999 generation faults: A review and analysis","volume":"2","author":"Duncan","year":"2018","journal-title":"J. Cyber Secur. Technol."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Tavallaee, M., Bagheri, E., Lu, W., and Ghorbani, A.A. (2009, January 8\u201310). A detailed analysis of the KDD CUP 99 data set. Proceedings of the 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, Ottawa, ON, Canada.","DOI":"10.1109\/CISDA.2009.5356528"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Massicotte, F., Gagnon, F., Labiche, Y., Briand, L., and Couture, M. (2006, January 11\u201315). Automatic evaluation of intrusion detection systems. Proceedings of the 2006 22nd Annual Computer Security Applications Conference (ACSAC\u201906), Miami Beach, FL, USA.","DOI":"10.1109\/ACSAC.2006.15"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1145\/382912.382923","article-title":"Testing intrusion detection systems: A critique of the 1998 and 1999 darpa intrusion detection system evaluations as performed by lincoln laboratory","volume":"3","author":"McHugh","year":"2000","journal-title":"ACM Trans. Inf. Syst. Secur."},{"key":"ref_5","unstructured":"(2022, March 19). A Realistic Cyberdefense Dataset (CSE-CIC-IDS2018). Available online: https:\/\/registry.opendata.aws\/cse-cic-ids2018\/."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Thapa, N., Liu, Z., Kc, D.B., Gokaraju, B., and Roy, K. (2020). Comparison of machine learning and deep learning models for network intrusion detection systems. Future Internet, 12.","DOI":"10.3390\/fi12100167"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1007\/978-981-16-0422-5_11","article-title":"Benchmarking behaviour-Based Intrusion Detection Systems with Bio-inspired Algorithms","volume":"Volume 1364","author":"Ferreira","year":"2021","journal-title":"Proceedings of the Security in Computing and Communications: 8th International Symposium, SSCC 2020"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40537-020-00382-x","article-title":"A survey and analysis of intrusion detection models based on cse-cic-ids2018 big data","volume":"7","author":"Leevy","year":"2020","journal-title":"J. Big Data"},{"key":"ref_9","first-page":"102419","article-title":"Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study","volume":"50","author":"Ferrag","year":"2020","journal-title":"J. Inf. Secur. Appl."},{"key":"ref_10","first-page":"1","article-title":"Towards Detecting and Classifying Network Intrusion Traffic Using Deep Learning Frameworks","volume":"9","author":"Basnet","year":"2019","journal-title":"J. Internet Serv. Inf. Secur."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Le, T.T.H., Kim, Y., and Kim, H. (2019). Network intrusion detection based on novel feature selection model and various recurrent neural networks. Appl. Sci., 9.","DOI":"10.3390\/app9071392"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1016\/j.gltp.2021.08.017","article-title":"Evaluating neural networks using Bi-Directional LSTM for network IDS (intrusion detection systems) in cyber security","volume":"2","author":"Pooja","year":"2021","journal-title":"Glob. Transitions Proc."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ding, Y., and Zhai, Y. (2018, January 8\u201310). Intrusion detection system for NSL-KDD dataset using convolutional neural networks. Proceedings of the 2018 2nd International Conference on Computer Science and Artificial Intelligence, Shenzhen, China.","DOI":"10.1145\/3297156.3297230"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.istr.2005.08.001","article-title":"Intrusion detection systems and intrusion prevention systems","volume":"10","author":"Fuchsberger","year":"2005","journal-title":"Inf. Secur. Tech. Rep."},{"key":"ref_15","unstructured":"Hindy, H., Brosset, D., Bayne, E., Seeam, A., Tachtatzis, C., Atkinson, R., and Bellekens, X. (2018). A taxonomy and survey of intrusion detection system design techniques, network threats and datasets. arXiv."},{"key":"ref_16","unstructured":"(2022, March 19). Snort\u2014Network Intrusion Detection & Prevention System. Available online: https:\/\/www.snort.org\/."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1109\/COMST.2018.2847722","article-title":"A detailed investigation and analysis of using machine learning techniques for intrusion detection","volume":"21","author":"Mishra","year":"2018","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Alsoufi, M.A., Razak, S., Siraj, M.M., Nafea, I., Ghaleb, F.A., Saeed, F., and Nasser, M. (2021). Anomaly-based intrusion detection systems in iot using deep learning: A systematic literature review. Appl. Sci., 11.","DOI":"10.3390\/app11188383"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mirza, A.H., and Cosan, S. (2018, January 2\u20135). Computer network intrusion detection using sequential LSTM neural networks autoencoders. Proceedings of the 2018 26th signal processing and communications applications conference (SIU), Izmir, Turkey.","DOI":"10.1109\/SIU.2018.8404689"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"132306","DOI":"10.1016\/j.physd.2019.132306","article-title":"Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network","volume":"404","author":"Sherstinsky","year":"2020","journal-title":"Phys. D Nonlinear Phenom."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Susilo, B., and Sari, R.F. (2020). Intrusion detection in IoT networks using deep learning algorithm. Information, 11.","DOI":"10.3390\/info11050279"},{"key":"ref_22","unstructured":"Patterson, J., and Gibson, A. (2017). Deep Learning: A practitioner\u2019s Approach, O\u2019Reilly Media, Inc."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kim, J., Kim, J., Kim, H., Shim, M., and Choi, E. (2020). CNN-based network intrusion detection against denial-of-service attacks. Electronics, 9.","DOI":"10.3390\/electronics9060916"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"12133","DOI":"10.1088\/1742-6596\/1353\/1\/012133","article-title":"Method of analyzing computer traffic based on recurrent neural networks","volume":"1353","author":"Chastikova","year":"2019","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Lin, P., Ye, K., and Xu, C.Z. (2019). Dynamic network anomaly detection system by using deep learning techniques. Cloud Computing\u2014CLOUD 2019, Proceedings of the International Conference on Cloud Computing, San Diego, CA, USA, 25\u201330 June 2019, Springer.","DOI":"10.1007\/978-3-030-23502-4_12"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"949","DOI":"10.1007\/s10586-017-1117-8","article-title":"A survey of deep-learning-based network anomaly detection","volume":"22","author":"Kwon","year":"2019","journal-title":"Clust. Comput."},{"key":"ref_27","unstructured":"Pinaya, W.H.L., Vieira, S., Garcia-Dias, R., and Mechelli, A. (2020). Autoencoders. Machine Learning, Elsevier."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Varma, P.R.K., Kumari, V.V., and Kumar, S.S. (2018). A survey of feature selection techniques in intrusion detection system: A soft computing perspective. Progress in Computing, Analytics and Networking, Springer.","DOI":"10.1007\/978-981-10-7871-2_75"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1080\/02564602.2020.1740615","article-title":"PCA-based feature reduction for hyperspectral remote sensing image classification","volume":"38","author":"Uddin","year":"2021","journal-title":"IETE Tech. Rev."},{"key":"ref_30","first-page":"108","article-title":"Toward generating a new intrusion detection dataset and intrusion traffic characterization","volume":"1","author":"Sharafaldin","year":"2018","journal-title":"ICISSp"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1016\/j.icte.2020.08.005","article-title":"A deep learning based HTTP slow DoS classification approach using flow data","volume":"7","author":"Muraleedharan","year":"2021","journal-title":"ICT Express"},{"key":"ref_32","unstructured":"(2022, March 19). Patator\u2014Penetration Testing Tools. Available online: https:\/\/en.kali.tools\/?p=147."},{"key":"ref_33","unstructured":"(2022, March 19). DVWA\u2014Damn Vulnerable Web Application. Available online: https:\/\/dvwa.co.uk\/."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Shah, M., Ahmed, S., Saeed, K., Junaid, M., Khan, H., and Rehman, A.U. (2019, January 30\u201331). Penetration testing active reconnaissance phase\u2013optimized port scanning with nmap tool. Proceedings of the 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), Sukkur, Pakistan.","DOI":"10.1109\/ICOMET.2019.8673520"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kompougias, O., Papadopoulos, D., Mantas, E., Litke, A., Papadakis, N., Paraschos, D., Kourtis, A., and Xylouris, G. (2021, January 7\u201310). IoT Botnet Detection on Flow Data using Autoencoders. Proceedings of the 2021 IEEE International Mediterranean Conference on Communications and Networking (MeditCom), Athens, Greece.","DOI":"10.1109\/MeditCom49071.2021.9647639"},{"key":"ref_36","unstructured":"Nagpal, B., Sharma, P., Chauhan, N., and Panesar, A. (2015, January 11\u201313). DDoS tools: Classification, analysis and comparison. Proceedings of the 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India."},{"key":"ref_37","unstructured":"(2022, March 19). Orange Data Mining\u2014Data Mining. Available online: https:\/\/orangedatamining.com\/."},{"key":"ref_38","unstructured":"(2022, March 19). Keras: The Python Deep Learning API. Available online: https:\/\/keras.io\/."},{"key":"ref_39","unstructured":"(2022, March 19). Tensorflow. Available online: https:\/\/tensorflow.org\/."},{"key":"ref_40","unstructured":"(2022, March 19). Scikit-Learn: Machine Learning in Python: Scikit-Lear 1.0.1. Available online: https:\/\/scikit-learn.org\/."},{"key":"ref_41","unstructured":"(2022, March 19). Matplotlib\u2014Visualization with Python. Available online: https:\/\/matplotlib.org\/."},{"key":"ref_42","first-page":"102564","article-title":"Inter-dataset generalization strength of supervised machine learning methods for intrusion detection","volume":"54","author":"Wauters","year":"2020","journal-title":"J. Inf. Secur. Appl."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Catillo, M., Rak, M., and Villano, U. (2020). 2l-zed-ids: A two-level anomaly detector for multiple attack classes. Artificial Intelligence and Network Applications\u2014WAINA 2020, Proceedings of the Workshops of the International Conference on Advanced Information Networking and Applications, Caserta, Italy, 15\u201317 April 2020, Springer.","DOI":"10.1007\/978-3-030-44038-1_63"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Huancayo Ramos, K.S., Sotelo Monge, M.A., and Maestre Vidal, J. (2020). Benchmark-based reference model for evaluating botnet detection tools driven by traffic-flow analytics. Sensors, 20.","DOI":"10.3390\/s20164501"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Fitni, Q.R.S., and Ramli, K. (2020, January 7\u20138). Implementation of ensemble learning and feature selection for performance improvements in anomaly-based intrusion detection systems. Proceedings of the 2020 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT), Bali, Indonesia.","DOI":"10.1109\/IAICT50021.2020.9172014"}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/9\/1\/29\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,27]],"date-time":"2024-07-27T16:37:26Z","timestamp":1722098246000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/9\/1\/29"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,20]]},"references-count":45,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["informatics9010029"],"URL":"https:\/\/doi.org\/10.3390\/informatics9010029","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,20]]}}}