Abstract
Vehicular networks are networks of communicating vehicles, a major enabling technology for future cooperative and autonomous driving technologies. The most important messages in these networks are broadcast-authenticated periodic one-hop beacons, used for safety and traffic efficiency applications such as collision avoidance and traffic jam detection. However, broadcast authenticity is not sufficient to guarantee message correctness. The goal of misbehavior detection is to analyze application data and knowledge about physical processes in these cyber-physical systems to detect incorrect messages, enabling local revocation of vehicles transmitting malicious messages. Comparative studies between detection mechanisms are rare due to the lack of a reference dataset. We take the first steps to address this challenge by introducing the Vehicular Reference Misbehavior Dataset (VeReMi) and a discussion of valid metrics for such an assessment. VeReMi is the first public extensible dataset, allowing anyone to reproduce the generation process, as well as contribute attacks and use the data to compare new detection mechanisms against existing ones. The result of our analysis shows that the acceptance range threshold and the simple speed check are complementary mechanisms that detect different attacks. This supports the intuitive notion that fusion can lead to better results with data, and we suggest that future work should focus on effective fusion with VeReMi as an evaluation baseline.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
For illustration purposes; our 100 s excerpt of the scenario at high densities contains hundreds of vehicles and runs for a few hours – a significant part of this cost is the realistic simulation of signals bouncing off the ground and various buildings.
- 3.
- 4.
References
Cárdenas, A.A., Baras, J.S., Seamon, K.: A framework for the evaluation of intrusion detection systems. In IEEE Symposium on Security and Privacy, p. 15–pp. IEEE (2006)
Codeca, L., Frank, R., Faye, S., Engel, T.: Luxembourg SUMO traffic (LuST) scenario: traffic demand evaluation. IEEE Intell. Transp. Syst. Mag. 9(2), 52–63 (2017)
Damgaard, C.: Gini coefficient. From MathWord - A Wolfram Web Resource, Created by Eric W. Weisstein. http://mathworld.wolfram.com/GiniCoefficient.html. Accessed 9 Feb 2018
Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, pp. 233–240. ACM, New York (2006)
Dietzel, S., van der Heijden, R.W., Decke, H., Kargl, F.: A flexible, subjective logic-based framework for misbehavior detection in V2V networks. Proceeding of IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks 2014, pp. 1–6 (2014)
Eberz, S., Rasmussen, K.B., Lenders, V., Martinovic, I.: Evaluating behavioral biometrics for continuous authentication: challenges and metrics. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, ASIA CCS 2017, pp. 386–399. ACM, New York (2017)
Jimenez, I., Sevilla, M., Watkins, N., Maltzahn, C., Lofstead, J., Mohror, K., Arpaci-Dusseau, A., Arpaci-Dussea, R.: The popper convention: making reproducible systems evaluation practical. In: 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 1561–1570, May 2017
Joerer, S., Sommer, C., Dressler, F.: Toward reproducibility and comparability of IVC simulation studies: a literature survey. IEEE Commun. Mag. 50(10), 82–88 (2012)
Kumar, V., Petit, J., Whyte, W.: Binary hash tree based certificate access management for connected vehicles. In: Proceedings of the 10th ACM Conference on Security and Privacy in Wireless and Mobile Networks, WiSec 2017, pp. 145–155. ACM, New York (2017)
Leinmüller, T., Schoch, E., Kargl, F., Maihöfer, C.: Decentralized position verification in geographic ad hoc routing. Security Commun. Netw. 3, 289–302 (2008)
Lo, N.-W., Tsai, H.-C.: Illusion attack on VANET applications-a message plausibility problem. In: 2007 IEEE Globecom Workshops, pp. 1–8. IEEE (2007)
Mitchell, R., Chen, I.-R.: A survey of intrusion detection techniques for cyber-physical systems. ACM Comput. Surv. 46(4), 55:1–55:29 (2014)
Raya, M., Papadimitratos, P., Gligor, V.D., Hubaux, J.P.: On data-centric trust establishment in ephemeral ad hoc networks. In: IEEE INFOCOM 2008 - The 27th Conference on Computer Communications, April 2008
Saini, M., Alelaiwi, A., Saddik, A.E.: How close are we to realizing a pragmatic VANET solution? a meta-survey. ACM Comput. Surv. 48(2), 29:1–29:40 (2015)
Saito, T., Rehmsmeier, M.: The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10(3), 1–21 (2015)
Schmidt, R.K., Leinmueller, T., Schoch, E., Held, A., Schaefer, G.: Vehicle behavior analysis to enhance security in VANETs. In: Proceedings of the 4th IEEE Vehicle-to-Vehicle Communications Workshop (V2VCOM2008) (2008)
Sommer, C., German, R., Dressler, F.: Bidirectionally coupled network and road traffic simulation for improved IVC analysis. IEEE Trans. Mob. Comput. 10(1), 3–15 (2011)
Stübing, H., Firl, J., Huss, S.A.: A two-stage verification process for Car-to-X mobility data based on path prediction and probabilistic maneuver recognition. In: 2011 IEEE Vehicular Networking Conference (VNC), pp. 17–24. IEEE (2011)
van der Heijden, R.W., Al-Momani, A., Kargl, F., Abu-Sharkh, O.M.F.: Enhanced position verification for VANETs using subjective logic. In: 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall), pp. 1–7, September 2016
van der Heijden, R.W., Dietzel, S., Leinmüller, T., Kargl, T.: Survey on misbehavior detection in cooperative intelligent transportation systems (2016). Arxiv Pre-Print http://arxiv.org/abs/1610.06810. Accessed 9 Feb 2018
van der Heijden, R.W., Kargl, F.: Evaluating misbehavior detection for vehicular networks. In: 5th GI/ITG KuVS FG Inter-vehicle Communication, p. 5 (2017)
van der Heijden, R.W., Lukaseder, T., Kargl, F.: Analyzing attacks on cooperative adaptive cruise control (CACC). In: 2017 IEEE Vehicular Networking Conference (VNC), pp. 45–52, November 2017
Acknowledgement
The authors thank Florian Diemer and Leo Hnatek for the contribution of several detector implementations in Maat, and Henning Kopp for discussions regarding the Gini index. Experiments for this work were performed on the computational resource bwUniCluster funded by the Ministry of Science, Research and the Arts Baden-Württemberg and the Universities of the State of Baden-Württemberg, Germany, within the framework program bwHPC. This work was supported in part by the Baden-Württemberg Stiftung gGmbH Stuttgart as part of the project IKT-05 AutoDetect of its IT security research programme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
van der Heijden, R.W., Lukaseder, T., Kargl, F. (2018). VeReMi: A Dataset for Comparable Evaluation of Misbehavior Detection in VANETs. In: Beyah, R., Chang, B., Li, Y., Zhu, S. (eds) Security and Privacy in Communication Networks. SecureComm 2018. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 254. Springer, Cham. https://doi.org/10.1007/978-3-030-01701-9_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-01701-9_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01700-2
Online ISBN: 978-3-030-01701-9
eBook Packages: Computer ScienceComputer Science (R0)