Intersection Focused Situation Coverage-Based Verification and Validation Framework for Autonomous Vehicles Implemented in CARLA | SpringerLink
Skip to main content
Springer Nature Link
Log in

Intersection Focused Situation Coverage-Based Verification and Validation Framework for Autonomous Vehicles Implemented in CARLA

  • Conference paper
  • First Online:
Modelling and Simulation for Autonomous Systems (MESAS 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13207))

Abstract

Autonomous Vehicles (AVs) i.e., self-driving cars, operate in a safety-critical domain, since errors in the autonomous driving software can lead to huge losses. Statistically, road intersections which are a part of the AVs operational design domain (ODD), have some of the highest accident rates. Hence, testing AVs to the limits on road intersections and assuring their safety on road intersections is pertinent, and thus the focus of this paper. We present a situation coverage-based (SitCov) AV-testing framework for the verification and validation (V&V) and safety assurance of AVs, developed in an open-source AV simulator named CARLA. The SitCov AV-testing framework focuses on vehicle-to-vehicle (V2V) interaction on a road intersection under different environmental conditions and intersection configuration situations (start/goal locations), using situation coverage criteria for automatic test suite generation for safety assurance of AVs. We have developed an ontology for intersection situations, and used it to generate a situation hyperspace i.e., the space of all possible situations arising from that ontology. For the evaluation of our SitCov AV-testing framework, we have seeded multiple faults in our ego AV, and compared situation coverage-based and random situation generation. We have found that both generation methodologies trigger around the same number of seeded faults, but the situation coverage-based generation tells us a lot more about the weaknesses of the autonomous driving algorithm of our ego AV, especially in edge-cases. Our code is publicly available online and since the simulation software (CARLA) is open-source, anyone can use our SitCov AV-testing framework and use it or build further on top of it. This paper aims to contribute to the domain of V&V and development of AVs, not only from a theoretical point of view, but also from the viewpoint of an open-source software contribution and releasing a flexible/effective tool for V&V and development of AVs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Tahir, Z., Alexander, R.: Coverage based testing for V&V and safety assurance of self-driving autonomous vehicles: a systematic literature review. In: 2020 IEEE International Conference On Artificial Intelligence Testing (AITest), pp. 23–30 (2020). https://doi.org/10.1109/AITEST49225.2020.00011

  2. International Organization for Standardization, ISO/PAS 21448:2019 Road vehicles—Safety of the intended functionality (2019). https://www.iso.org/standard/70939

  3. International Organization for Standardization, ISO 26262-1:2018 Road vehicles—Functional safety (2018). https://www.iso.org/standard/68383

  4. Underwriters Laboratories, Presenting the Standard for Safety for the Evaluation of Autonomous Vehicles and Other Products. https://ul.org/UL4600

  5. Anthony, C., Lee, R., Kochenderfer, M.J.: Scalable autonomous vehicle safety validation through dynamic programming and scene decomposition. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE (2020)

    Google Scholar 

  6. Klischat, M., Althoff, M.: Generating critical test scenarios for automated vehicles with evolutionary algorithms. In: 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE (2019)

    Google Scholar 

  7. Chance, G., Ghobrial, A., Lemaignan, S., Pipe, T., Eder, K.: An agency-directed approach to test generation for simulation-based autonomous vehicle verification. In: 2020 IEEE International Conference on Artificial Intelligence Testing (AITest). IEEE (2020)

    Google Scholar 

  8. Haq, F.U., Shin, D., Nejati, S., Briand, L.: Comparing offline and online testing of deep neural networks: An autonomous car case study. In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE (2020)

    Google Scholar 

  9. Society of Automotive Engineers, SAE J-3016 international report at https://www.sae.org

  10. MathWorks Automated Driving Toolbox. https://uk.mathworks.com/products/automated-driving.html

  11. CarMaker: Virtual testing of automobiles and light-duty vehicles. https://ipg-automotive.com/products-services/simulation-software/carmaker/

  12. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Conference on Robot Learning. PMLR (2017)

    Google Scholar 

  13. Kitchenham, B.A., et al.: Refining the systematic literature review process—two participant-observer case studies. Empir. Softw. Eng. 15(6), 618–653 (2010). https://doi.org/10.1007/s10664-010-9134-8

    Article  Google Scholar 

  14. Alexander, R., Hawkins, H., Rae, D.: Situation coverage – a coverage criterion for testing autonomous robots, pp. 1–20 (2015)

    Google Scholar 

  15. Ulbrich, S., Menzel, T., Reschka, A., Schuldt, F., Maurer, M.: Defining and substantiating the terms scene, situation, and scenario for automated driving. In: IEEE Conference Intelligent Transportation System Proceedings, ITSC, vol. 2015-Octob, pp. 982–988 (2015)

    Google Scholar 

  16. Tahir, Z.: Situation hyperspace—using a simulated world to obtain situation coverage for AV safety assurance. https://assuringautonomy.medium.com/situation-hyperspace-using-a-simulated-world-to-obtain-situation-coverage-for-av-safety-assurance-39fa5ea203cd

  17. Krzysztof, C.: Operational world model ontology for automated driving systems - part 1: road structure. https://doi.org/10.13140/RG.2.2.15521.30568

  18. Thorn, E., Kimmel, S., Chaka, M.: A framework for automated driving system testable cases and scenarios. National Highway Traffic Safety Administration USA (2018)

    Google Scholar 

  19. Philippe, N.: Safety-critical scenarios and virtual testing procedures for automated cars at road intersections. Diss. Loughborough University (2018)

    Google Scholar 

  20. Tahir, Z.: Situation Coverage-based AV-Testing Framework in Carla. https://github.com/zaidtahirbutt/Situation-Coverage-based-AV-Testing-Framework-in-CARLA

  21. International Organization for Standardization, ISO 22737:2021, Intelligent transport systems—Low-speed automated driving (LSAD) systems for predefined routes—Performance requirements, system requirements and performance test procedures. https://www.iso.org/standard/73767

  22. Xueyi, Z., Alexander, R., McDermid, J.: Testing method for multi-UAV conflict resolution using agent-based simulation and multi-objective search. J. Aerosp. Inf. Syst. 13(5), 191–203 (2016)

    Google Scholar 

  23. Krzysztof, C.: Operational world model ontology for automated driving systems - part 2: road users, animals, other obstacles, and environmental conditions. https://doi.org/10.13140/RG.2.2.11327.00165

  24. Krzysztof, C.: Operational design domain for automated driving systems - taxonomy of basic terms. https://doi.org/10.13140/RG.2.2.18037.88803

  25. Zhu, H., Hall, P.A.V., May, J.H.R.: Software unit test coverage and adequacy. ACM Comput. Surv. 29(4), 366–427 (1997). https://doi.org/10.1145/267580.267590

    Article  Google Scholar 

  26. Shwartz, S.S., Shammah, S., Shashua, A.: On a formal model of safe and scalable self-driving cars. https://arxiv.org/abs/1708.06374

  27. ScenarioRunner for CARLA. https://github.com/carla-simulator/scenario_runner

  28. Jonathan, H., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

    Google Scholar 

  29. LaValle, S.M.: Planning Algorithms. Cambridge University Press, Cambridge (2006)

    Book  Google Scholar 

  30. COCO dataset. https://cocodataset.org/

  31. Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. ArXiv preprint arXiv:1704.04861 (2017)

  32. Kinsley, H.: Object detection with Tensorflow - Self Driving Cars p. 17. https://www.youtube.com/watch?v=UAXulqzn5Ps

  33. Babikian, A.A.: Automated generation of test scenario models for the system-level safety assurance of autonomous vehicles. In: Proceedings of the 23rd ACM/IEEE MODELS (2020)

    Google Scholar 

Download references

Acknowledgements

The research presented in this paper has been funded by European Union’s EU Framework Programme for Research and Innovation Horizon 2020 under Grant Agreement No. 812.788.

Author information

Authors and Affiliations

  1. Assuring Autonomy International Programme, Department of Computer Science, University of York, York, UK

    Zaid Tahir

  2. ECE Department, Boston University, Boston, USA

    Zaid Tahir & Rob Alexander

Authors
  1. Zaid Tahir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rob Alexander
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zaid Tahir .

Editor information

Editors and Affiliations

  1. NATO M&S COE, Rome, Italy

    Jan Mazal

  2. University of Palermo, Palermo, Italy

    Adriano Fagiolini

  3. Brno University of Technology, Brno, Czech Republic

    Petr Vasik

  4. NATO M&S COE, Rome, Italy

    Michele Turi

  5. Savona Campus, University of Genoa, Savona, Italy

    Agostino Bruzzone

  6. Bundeswehr University Munich, Neubiberg, München, Germany

    Stefan Pickl

  7. University of Defence, Brno, Czech Republic

    Vlastimil Neumann

  8. University of Defence, Brno, Czech Republic

    Petr Stodola

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tahir, Z., Alexander, R. (2022). Intersection Focused Situation Coverage-Based Verification and Validation Framework for Autonomous Vehicles Implemented in CARLA. In: Mazal, J., et al. Modelling and Simulation for Autonomous Systems. MESAS 2021. Lecture Notes in Computer Science, vol 13207. Springer, Cham. https://doi.org/10.1007/978-3-030-98260-7_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-98260-7_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-98259-1

  • Online ISBN: 978-3-030-98260-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation